Deepfake Histologic Images for Enhancing Digital Pathology.

A pathologist's optical microscopic examination of thinly cut, stained tissue on glass slides prepared from a formalin-fixed paraffin-embedded tissue blocks is the gold standard for tissue diagnostics. In addition, the diagnostic abilities and expertise of pathologists is dependent on their direct experience with common and rarer variant morphologies. Recently, deep learning approaches have been used to successfully show a high level of accuracy for such tasks. However, obtaining expert-level annotated images is an expensive and time-consuming task, and artificially synthesized histologic images can prove greatly beneficial. In this study, we present an approach to not only generate histologic images that reproduce the diagnostic morphologic features of common disease but also provide a user ability to generate new and rare morphologies. Our approach involves developing a generative adversarial network model that synthesizes pathology images constrained by class labels. We investigated the ability of this framework in synthesizing realistic prostate and colon tissue images and assessed the utility of these images in augmenting the diagnostic ability of machine learning methods and their usability by a panel of experienced anatomic pathologists. Synthetic data generated by our framework performed similar to real data when training a deep learning model for diagnosis. Pathologists were not able to distinguish between real and synthetic images, and their analyses showed a similar level of interobserver agreement for prostate cancer grading. We extended the approach to significantly more complex images from colon biopsies and showed that the morphology of the complex microenvironment in such tissues can be reproduced. Finally, we present the ability for a user to generate deepfake histologic images using a simple markup of sematic labels.

All-digital histopathology by infrared-optical hybrid microscopy.

Optical microscopy for biomedical samples requires expertise in staining to visualize structure and composition. Midinfrared (mid-IR) spectroscopic imaging offers label-free molecular recording and virtual staining by probing fundamental vibrational modes of molecular components. This quantitative signal can be combined with machine learning to enable microscopy in diverse fields from cancer diagnoses to forensics. However, absorption of IR light by common optical imaging components makes mid-IR light incompatible with modern optical microscopy and almost all biomedical research and clinical workflows. Here we conceptualize an IR-optical hybrid (IR-OH) approach that sensitively measures molecular composition based on an optical microscope with wide-field interferometric detection of absorption-induced sample expansion. We demonstrate that IR-OH exceeds state-of-the-art IR microscopy in coverage (10-fold), spatial resolution (fourfold), and spectral consistency (by mitigating the effects of scattering). The combined impact of these advances allows full slide infrared absorption images of unstained breast tissue sections on a visible microscope platform. We further show that automated histopathologic segmentation and generation of computationally stained (stainless) images is possible, resolving morphological features in both color and spatial detail comparable to current pathology protocols but without stains or human interpretation. IR-OH is compatible with clinical and research pathology practice and could make for a cost-effective alternative to conventional stain-based protocols for stainless, all-digital pathology.

A generative adversarial approach to facilitate archival-quality histopathologic diagnoses from frozen tissue sections.

In clinical diagnostics and research involving histopathology, formalin-fixed paraffin-embedded (FFPE) tissue is almost universally favored for its superb image quality. However, tissue processing time (>24 h) can slow decision-making. In contrast, fresh frozen (FF) processing (<1 h) can yield rapid information but diagnostic accuracy is suboptimal due to lack of clearing, morphologic deformation and more frequent artifacts. Here, we bridge this gap using artificial intelligence. We synthesize FFPE-like images ("virtual FFPE") from FF images using a generative adversarial network (GAN) from 98 paired kidney samples derived from 40 patients. Five board-certified pathologists evaluated the results in a blinded test. Image quality of the virtual FFPE data was assessed to be high and showed a close resemblance to real FFPE images. Clinical assessments of disease on the virtual FFPE images showed a higher inter-observer agreement compared to FF images. The nearly instantaneously generated virtual FFPE images can not only reduce time to information but can facilitate more precise diagnosis from routine FF images without extraneous costs and effort.

INSTRAS: INfrared Spectroscopic imaging-based TRAnsformers for medical image Segmentation.

Infrared (IR) spectroscopic imaging is of potentially wide use in medical imaging applications due to its ability to capture both chemical and spatial information. This complexity of the data both necessitates using machine intelligence as well as presents an opportunity to harness a high-dimensionality data set that offers far more information than today's manually-interpreted images. While convolutional neural networks (CNNs), including the well-known U-Net model, have demonstrated impressive performance in image segmentation, the inherent locality of convolution limits the effectiveness of these models for encoding IR data, resulting in suboptimal performance. In this work, we propose an INfrared Spectroscopic imaging-based TRAnsformers for medical image Segmentation (INSTRAS). This novel model leverages the strength of the transformer encoders to segment IR breast images effectively. Incorporating skip-connection and transformer encoders, INSTRAS overcomes the issue of pure convolution models, such as the difficulty of capturing long-range dependencies. To evaluate the performance of our model and existing convolutional models, we conducted training on various encoder-decoder models using a breast dataset of IR images. INSTRAS, utilizing 9 spectral bands for segmentation, achieved a remarkable AUC score of 0.9788, underscoring its superior capabilities compared to purely convolutional models. These experimental results attest to INSTRAS's advanced and improved segmentation abilities for IR imaging.

Rapid and Label-Free Histopathology of Oral Lesions Using Deep Learning Applied to Optical and Infrared Spectroscopic Imaging Data.

Oral potentially malignant disorders (OPMDs) are precursors to over 80% of oral cancers. Hematoxylin and eosin (H&E) staining, followed by pathologist interpretation of tissue and cellular morphology, is the current gold standard for diagnosis. However, this method is qualitative, can result in errors during the multi-step diagnostic process, and results may have significant inter-observer variability. Chemical imaging (CI) offers a promising alternative, wherein label-free imaging is used to record both the morphology and the composition of tissue and artificial intelligence (AI) is used to objectively assign histologic information. Here, we employ quantum cascade laser (QCL)-based discrete frequency infrared (DFIR) chemical imaging to record data from oral tissues. In this proof-of-concept study, we focused on achieving tissue segmentation into three classes (connective tissue, dysplastic epithelium, and normal epithelium) using a convolutional neural network (CNN) applied to three bands of label-free DFIR data with paired darkfield visible imaging. Using pathologist-annotated H&E images as the ground truth, we demonstrate results that are 94.5% accurate with the ground truth using combined information from IR and darkfield microscopy in a deep learning framework. This chemical-imaging-based workflow for OPMD classification has the potential to enhance the efficiency and accuracy of clinical oral precancer diagnosis.

Accelerating Cancer Histopathology Workflows with Chemical Imaging and Machine Learning.

Histopathology has remained a cornerstone for biomedical tissue assessment for over a century, with a resource-intensive workflow involving biopsy or excision, gross examination, sampling, tissue processing to snap frozen or formalin-fixed paraffin-embedded blocks, sectioning, staining, optical imaging, and microscopic assessment. Emerging chemical imaging approaches, including stimulated Raman scattering (SRS) microscopy, can directly measure inherent molecular composition in tissue (thereby dispensing with the need for tissue processing, sectioning, and using dyes) and can use artificial intelligence (AI) algorithms to provide high-quality images. Here we show the integration of SRS microscopy in a pathology workflow to rapidly record chemical information from minimally processed fresh-frozen prostate tissue. Instead of using thin sections, we record data from intact thick tissues and use optical sectioning to generate images from multiple planes. We use a deep learning­based processing pipeline to generate virtual hematoxylin and eosin images. Next, we extend the computational method to generate archival-quality images in minutes, which are equivalent to those obtained from hours/days-long formalin-fixed, paraffin-embedded processing. We assessed the quality of images from the perspective of enabling pathologists to make decisions, demonstrating that the virtual stained image quality was diagnostically useful and the interpathologist agreement on prostate cancer grade was not impacted. Finally, because this method does not wash away lipids and small molecules, we assessed the utility of lipid chemical composition in determining grade. Together, the combination of chemical imaging and AI provides novel capabilities for rapid assessments in pathology by reducing the complexity and burden of current workflows. SIGNIFICANCE: Archival-quality (formalin-fixed paraffin-embedded), thin-section diagnostic images are obtained from thick-cut, fresh-frozen prostate tissues without dyes or stains to expedite cancer histopathology by combining SRS microscopy and machine learning.

Infrared spectroscopic laser scanning confocal microscopy for whole-slide chemical imaging.

Chemical imaging, especially mid-infrared spectroscopic microscopy, enables label-free biomedical analyses while achieving expansive molecular sensitivity. However, its slow speed and poor image quality impede widespread adoption. We present a microscope that provides high-throughput recording, low noise, and high spatial resolution where the bottom-up design of its optical train facilitates dual-axis galvo laser scanning of a diffraction-limited focal point over large areas using custom, compound, infinity-corrected refractive objectives. We demonstrate whole-slide, speckle-free imaging in ~3 min per discrete wavelength at 10× magnification (2 µm/pixel) and high-resolution capability with its 20× counterpart (1 µm/pixel), both offering spatial quality at theoretical limits while maintaining high signal-to-noise ratios (>100:1). The data quality enables applications of modern machine learning and capabilities not previously feasible - 3D reconstructions using serial sections, comprehensive assessments of whole model organisms, and histological assessments of disease in time comparable to clinical workflows. Distinct from conventional approaches that focus on morphological investigations or immunostaining techniques, this development makes label-free imaging of minimally processed tissue practical.

Colon Cancer Grading Using Infrared Spectroscopic Imaging-Based Deep Learning.

Tumor grade assessment is critical to the treatment of cancers. A pathologist typically evaluates grade by examining morphologic organization in tissue using hematoxylin and eosin (H&E) stained tissue sections. Fourier transform infrared spectroscopic (FT-IR) imaging provides an alternate view of tissue in which spatially specific molecular information from unstained tissue can be utilized. Here, we examine the potential of IR imaging for grading colon cancer in biopsy samples. We used a 148-patient cohort to develop a deep learning classifier to estimate the tumor grade using IR absorption. We demonstrate that FT-IR imaging can be a viable tool to determine colorectal cancer grades, which we validated on an independent cohort of surgical resections. This work demonstrates that harnessing molecular information from FT-IR imaging and coupling it with morphometry is a potential path to develop clinically relevant grade prediction models.

Detection and Quantification of Myocardial Fibrosis Using Stain-Free Infrared Spectroscopic Imaging.

CONTEXT.­: Myocardial fibrosis underpins a number of cardiovascular conditions and is difficult to identify with standard histologic techniques. Challenges include imaging, defining an objective threshold for classifying fibrosis as mild or severe, and understanding the molecular basis for these changes. OBJECTIVE.­: To develop a novel, rapid, label-free approach to accurately measure and quantify the extent of fibrosis in cardiac tissue using infrared spectroscopic imaging. DESIGN.­: We performed infrared spectroscopic imaging and combined that with advanced machine learning-based algorithms to assess fibrosis in 15 samples from patients belonging to the following 3 classes: (1) patients with nonpathologic (control) donor hearts, (2) patients undergoing transplant, and (3) patients undergoing implantation of a ventricular assist device. RESULTS.­: Our results show excellent sensitivity and accuracy for detecting myocardial fibrosis, as demonstrated by a high area under the curve of 0.998 in the receiver operating characteristic curve measured from infrared imaging. Fibrosis of various morphologic subtypes were demonstrated with virtually generated picrosirius red images, which showed good visual and quantitative agreement (correlation coefficient = 0.92, ρ = 7.76 × 10-15) with stained images of the same sections. Underlying molecular composition of the different subtypes was investigated with infrared spectra showing reproducible differences presumably arising from differences in collagen subtypes and/or crosslinking. CONCLUSIONS.­: Infrared imaging can be a powerful tool in studying myocardial fibrosis and gleaning insights into the underlying chemical changes that accompany it. Emerging methods suggest that the proposed approach is compatible with conventional optical microscopy, and its consistency makes it translatable to the clinical setting for real-time diagnoses as well as for objective and quantitative research.