Artificial intelligence-enabled quantitative phase imaging methods for life sciences.

Quantitative phase imaging, integrated with artificial intelligence, allows for the rapid and label-free investigation of the physiology and pathology of biological systems. This review presents the principles of various two-dimensional and three-dimensional label-free phase imaging techniques that exploit refractive index as an intrinsic optical imaging contrast. In particular, we discuss artificial intelligence-based analysis methodologies for biomedical studies including image enhancement, segmentation of cellular or subcellular structures, classification of types of biological samples and image translation to furnish subcellular and histochemical information from label-free phase images. We also discuss the advantages and challenges of artificial intelligence-enabled quantitative phase imaging analyses, summarize recent notable applications in the life sciences, and cover the potential of this field for basic and industrial research in the life sciences.

Bortezomib elevates intracellular free Fe2+ by enhancing NCOA4-mediated ferritinophagy and synergizes with RSL-3 to inhibit multiple myeloma cells.

Iron contributes to tumor initiation and progression; however, excessive intracellular free Fe2+ can be toxic to cancer cells. Our findings confirmed that multiple myeloma (MM) cells exhibited elevated intracellular iron levels and increased ferritin, a key protein for iron storage, compared with normal cells. Interestingly, Bortezomib (BTZ) was found to trigger ferritin degradation, increase free intracellular Fe2+, and promote ferroptosis in MM cells. Subsequent mechanistic investigation revealed that BTZ effectively increased NCOA4 levels by preventing proteasomal degradation in MM cells. When we knocked down NCOA4 or blocked autophagy using chloroquine, BTZ-induced ferritin degradation and the increase in intracellular free Fe2+ were significantly reduced in MM cells, confirming the role of BTZ in enhancing ferritinophagy. Furthermore, the combination of BTZ with RSL-3, a specific inhibitor of GPX4 and inducer of ferroptosis, synergistically promoted ferroptosis in MM cell lines and increased cell death in both MM cell lines and primary MM cells. The induction of ferroptosis inhibitor liproxstatin-1 successfully counteracted the synergistic effect of BTZ and RSL-3 in MM cells. Altogether, our findings reveal that BTZ elevates intracellular free Fe2+ by enhancing NCOA4-mediated ferritinophagy and synergizes with RSL-3 by increasing ferroptosisin MM cells.

Break Down in Order To Build Up: Decomposing Small Molecules for Fragment-Based Drug Design with eMolFrag.

Constructing high-quality libraries of molecular building blocks is essential for successful fragment-based drug discovery. In this communication, we describe eMolFrag, a new open-source software to decompose organic compounds into nonredundant fragments retaining molecular connectivity information. Given a collection of molecules, eMolFrag generates a set of unique fragments comprising larger moieties, bricks, and smaller linkers connecting bricks. These building blocks can subsequently be used to construct virtual screening libraries for targeted drug discovery. The robustness and computational performance of eMolFrag is assessed against the Directory of Useful Decoys, Enhanced database conducted in serial and parallel modes with up to 16 computing cores. Further, the application of eMolFrag in de novo drug design is illustrated using the adenosine receptor. eMolFrag is implemented in Python, and it is available as stand-alone software and a web server at www.brylinski.org/emolfrag and https://github.com/liutairan/eMolFrag .

Virtual histological staining of unlabeled autopsy tissue.

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

Rapid and stain-free quantification of viral plaque via lens-free holography and deep learning.

A plaque assay-the gold-standard method for measuring the concentration of replication-competent lytic virions-requires staining and usually more than 48 h of runtime. Here we show that lens-free holographic imaging and deep learning can be combined to expedite and automate the assay. The compact imaging device captures phase information label-free at a rate of approximately 0.32 gigapixels per hour per well, covers an area of about 30 × 30 mm2 and a 10-fold larger dynamic range of virus concentration than standard assays, and quantifies the infected area and the number of plaque-forming units. For the vesicular stomatitis virus, the automated plaque assay detected the first cell-lysing events caused by viral replication as early as 5 h after incubation, and in less than 20 h it detected plaque-forming units at rates higher than 90% at 100% specificity. Furthermore, it reduced the incubation time of the herpes simplex virus type 1 by about 48 h and that of the encephalomyocarditis virus by about 20 h. The stain-free assay should be amenable for use in virology research, vaccine development and clinical diagnosis.

Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization.

Deep learning-based image reconstruction methods have achieved remarkable success in phase recovery and holographic imaging. However, the generalization of their image reconstruction performance to new types of samples never seen by the network remains a challenge. Here we introduce a deep learning framework, termed Fourier Imager Network (FIN), that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting unprecedented success in external generalization. FIN architecture is based on spatial Fourier transform modules that process the spatial frequencies of its inputs using learnable filters and a global receptive field. Compared with existing convolutional deep neural networks used for hologram reconstruction, FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ~0.04 s per 1 mm2 of the sample area. We experimentally validated the performance of FIN by training it using human lung tissue samples and blindly testing it on human prostate, salivary gland tissue and Pap smear samples, proving its superior external generalization and image reconstruction speed. Beyond holographic microscopy and quantitative phase imaging, FIN and the underlying neural network architecture might open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.

Virtual Impactor-Based Label-Free Pollen Detection using Holography and Deep Learning.

Exposure to bio-aerosols such as pollen can lead to adverse health effects. There is a need for a portable and cost-effective device for long-term monitoring and quantification of various types of pollen. To address this need, we present a mobile and cost-effective label-free sensor that takes holographic images of flowing particulate matter (PM) concentrated by a virtual impactor, which selectively slows down and guides particles larger than 6 µm to fly through an imaging window. The flowing particles are illuminated by a pulsed laser diode, casting their inline holograms on a complementary metal-oxide semiconductor image sensor in a lens-free mobile imaging device. The illumination contains three short pulses with a negligible shift of the flowing particle within one pulse, and triplicate holograms of the same particle are recorded at a single frame before it exits the imaging field-of-view, revealing different perspectives of each particle. The particles within the virtual impactor are localized through a differential detection scheme, and a deep neural network classifies the pollen type in a label-free manner based on the acquired holographic images. We demonstrated the success of this mobile pollen detector with a virtual impactor using different types of pollen (i.e., bermuda, elm, oak, pine, sycamore, and wheat) and achieved a blind classification accuracy of 92.91%. This mobile and cost-effective device weighs ∼700 g and can be used for label-free sensing and quantification of various bio-aerosols over extended periods since it is based on a cartridge-free virtual impactor that does not capture or immobilize PM.

Distance + Angle-Based Control of 2-D Rigid Formations.

A well-known problem with distance-based formation control is the existence of multiple equilibrium points not associated with the desired formation. This problem can be potentially mitigated by introducing an additional controlled variable. In this article, we generalize the distance + angle-based scheme for 2-D formations of single-integrator agents by using directed graphs and triangulation of the n-agent formation. We show that under certain conditions on the control gains and desired formation shape, our controller ensures the asymptotic stability of the correct formation for almost all initial agent positions.

Radiation Mapping for an Unmanned Aerial Vehicle: Development and Simulated Testing of Algorithms for Source Mapping and Navigation Path Generation.

ABSTRACT: Image reconstruction algorithms were developed for radiation source mapping and used for generating the search path of a moving radiation detector, such as one onboard an unmanned aerial vehicle. Simulations consisted of first assuming radioactive sources of varying complexity and estimating the radiation fields that would then be produced by that source distribution. Next, the "measurements" that would result from a pair of adjacent spatial locations were computed. A crude estimate of the source distribution likely to have produced such "measurements" was reconstructed based upon the limited measurements. Location of the next "measurement" was then determined as halfway between the location of the estimated source and the current "measurement." With each additional sample, improved source distribution reconstructions were made and used to inform the immediate direction of detector motion. Source reconstruction or mapping was formulated as an inverse problem solved with either maximum a posteriori or least squares (LS) regression deconvolution methods. Different amounts of noise were added to the simulated "measurements," allowing evaluation of the methods' performances as functions of signal-to-noise ratio of the measured map. As expected, methods that promote sparsity were better suited in reconstructing point sources. Reliable prior information of the source distribution also improved the reconstruction results, especially with distributed sources. With a non-negative least square algorithm and the suggested paths it generated, location of sources was successfully estimated to an accuracy of 0.014 m within nine iterations in a single-source scenario and 12 iterations in a two-source scenario, given a 10% error on the integrated counts and a Poisson distribution of the noise associated with the measured counts.

Calcium pyrophosphate crystal size and characteristics.

OBJECTIVE: To describe the characteristics of calcium pyrophosphate (CPP) crystal size and morphology under compensated polarized light microscopy (CPLM). Secondarily, to describe CPP crystals seen only with digital enhancement of CPLM images, confirmed with advanced imaging techniques. METHODS: Clinical lab-identified CPP-positive synovial fluid samples were collected from 16 joint aspirates. Four raters used a standardized protocol to describe crystal shape, birefringence strength and color. A crystal expert confirmed CPLM-visualized crystal identification. For crystal measurement, a high-pass linear light filter was used to enhance resolution and line discrimination of digital images. This process identified additional enhanced crystals not seen by raters under CPLM. Single-shot computational polarized light microscopy (SCPLM) provided further confirmation of the enhanced crystals' presence. RESULTS: Of 932 suspected crystals identified by CPLM, 569 met our inclusion criteria, and 293 (51%) were confirmed as CPP crystals. Of 175 unique confirmed crystals, 118 (67%) were rods (median area 3.6 µm2 [range, 1.0-22.9 µm2]), and 57 (33%) were rhomboids (median area 4.8 µm2 [range, 0.9-16.7 µm2]). Crystals visualized only after digital image enhancement were smaller and less birefringent than CPLM-identified crystals. CONCLUSIONS: CPP crystals that are smaller and weakly birefringent are more difficult to identify. There is likely a population of smaller, less birefringent CPP crystals that routinely goes undetected by CPLM. Describing the characteristics of poorly visible crystals may be of use for future development of novel crystal identification methods.

Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data.

Optical coherence tomography (OCT) is a widely used non-invasive biomedical imaging modality that can rapidly provide volumetric images of samples. Here, we present a deep learning-based image reconstruction framework that can generate swept-source OCT (SS-OCT) images using undersampled spectral data, without any spatial aliasing artifacts. This neural network-based image reconstruction does not require any hardware changes to the optical setup and can be easily integrated with existing swept-source or spectral-domain OCT systems to reduce the amount of raw spectral data to be acquired. To show the efficacy of this framework, we trained and blindly tested a deep neural network using mouse embryo samples imaged by an SS-OCT system. Using 2-fold undersampled spectral data (i.e., 640 spectral points per A-line), the trained neural network can blindly reconstruct 512 A-lines in 0.59 ms using multiple graphics-processing units (GPUs), removing spatial aliasing artifacts due to spectral undersampling, also presenting a very good match to the images of the same samples, reconstructed using the full spectral OCT data (i.e., 1280 spectral points per A-line). We also successfully demonstrate that this framework can be further extended to process 3× undersampled spectral data per A-line, with some performance degradation in the reconstructed image quality compared to 2× spectral undersampling. Furthermore, an A-line-optimized undersampling method is presented by jointly optimizing the spectral sampling locations and the corresponding image reconstruction network, which improved the overall imaging performance using less spectral data points per A-line compared to 2× or 3× spectral undersampling results. This deep learning-enabled image reconstruction approach can be broadly used in various forms of spectral-domain OCT systems, helping to increase their imaging speed without sacrificing image resolution and signal-to-noise ratio.

Biopsy-free in vivo virtual histology of skin using deep learning.

An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool, it requires specialized training because the acquired images are grayscale, lack nuclear features, and are difficult to correlate with tissue pathology. Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers. The network was trained under an adversarial learning scheme, which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo, label-free RCM images of normal skin structure, basal cell carcinoma, and melanocytic nevi with pigmented melanocytes, demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.

Deep learning-based transformation of H&E stained tissues into special stains.

Pathology is practiced by visual inspection of histochemically stained tissue slides. While the hematoxylin and eosin (H&E) stain is most commonly used, special stains can provide additional contrast to different tissue components. Here, we demonstrate the utility of supervised learning-based computational stain transformation from H&E to special stains (Masson's Trichrome, periodic acid-Schiff and Jones silver stain) using kidney needle core biopsy tissue sections. Based on the evaluation by three renal pathologists, followed by adjudication by a fourth pathologist, we show that the generation of virtual special stains from existing H&E images improves the diagnosis of several non-neoplastic kidney diseases, sampled from 58 unique subjects (P = 0.0095). A second study found that the quality of the computationally generated special stains was statistically equivalent to those which were histochemically stained. This stain-to-stain transformation framework can improve preliminary diagnoses when additional special stains are needed, also providing significant savings in time and cost.

Pathological crystal imaging with single-shot computational polarized light microscopy.

Pathological crystal identification is routinely practiced in rheumatology for diagnosing arthritis disease such as gout, and relies on polarized light microscopy as the gold standard method used by medical professionals. Here, we present a single-shot computational polarized light microscopy method that reconstructs the transmittance, retardance and slow-axis orientation of a birefringent sample using a single image captured with a pixelated-polarizer camera. This method is fast, simple-to-operate and compatible with all the existing standard microscopes without extensive or costly modifications. We demonstrated the success of our method by imaging three different types of crystals found in synovial fluid and reconstructed the birefringence information of these samples using a single image, without being affected by the orientation of individual crystals within the sample field-of-view. We believe this technique will provide improved sensitivity, specificity and speed, all at low cost, for clinical diagnosis of crystals found in synovial fluid and other bodily fluids.

Deep Learning-Based Holographic Polarization Microscopy.

Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs, or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase-recovered hologram, while only requiring the addition of one polarizer/analyzer pair to an inline lensfree holographic imaging system. Using a deep neural network, the reconstructed holographic images from a single state of polarization can be transformed into images equivalent to those captured using a single-shot computational polarized light microscope (SCPLM). Our analysis shows that a trained deep neural network can extract the birefringence information using both the sample specific morphological features as well as the holographic amplitude and phase distribution. To demonstrate the efficacy of this method, we tested it by imaging various birefringent samples including, for example, monosodium urate and triamcinolone acetonide crystals. Our method achieves similar results to SCPLM both qualitatively and quantitatively, and due to its simpler optical design and significantly larger field-of-view this method has the potential to expand the access to polarization microscopy and its use for medical diagnosis in resource limited settings.

PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning.

Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform the quantitative phase images (QPI) of label-free tissue sections into images that are equivalent to the brightfield microscopy images of the same samples that are histologically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining), we train a generative adversarial network and demonstrate the effectiveness of this virtual-staining approach using sections of human skin, kidney, and liver tissue, matching the brightfield microscopy images of the same samples stained with Hematoxylin and Eosin, Jones' stain, and Masson's trichrome stain, respectively. This digital-staining framework may further strengthen various uses of label-free QPI techniques in pathology applications and biomedical research in general, by eliminating the need for histological staining, reducing sample preparation related costs and saving time. Our results provide a powerful example of some of the unique opportunities created by data-driven image transformations enabled by deep learning.

eToxPred: a machine learning-based approach to estimate the toxicity of drug candidates.

BACKGROUND: The efficiency of drug development defined as a number of successfully launched new pharmaceuticals normalized by financial investments has significantly declined. Nonetheless, recent advances in high-throughput experimental techniques and computational modeling promise reductions in the costs and development times required to bring new drugs to market. The prediction of toxicity of drug candidates is one of the important components of modern drug discovery. RESULTS: In this work, we describe eToxPred, a new approach to reliably estimate the toxicity and synthetic accessibility of small organic compounds. eToxPred employs machine learning algorithms trained on molecular fingerprints to evaluate drug candidates. The performance is assessed against multiple datasets containing known drugs, potentially hazardous chemicals, natural products, and synthetic bioactive compounds. Encouragingly, eToxPred predicts the synthetic accessibility with the mean square error of only 4% and the toxicity with the accuracy of as high as 72%. CONCLUSIONS: eToxPred can be incorporated into protocols to construct custom libraries for virtual screening in order to filter out those drug candidates that are potentially toxic or would be difficult to synthesize. It is freely available as a stand-alone software at https://github.com/pulimeng/etoxpred .

Accurate color imaging of pathology slides using holography and absorbance spectrum estimation of histochemical stains.

Holographic microscopy presents challenges for color reproduction due to the usage of narrow-band illumination sources, which especially impacts the imaging of stained pathology slides for clinical diagnoses. Here, an accurate color holographic microscopy framework using absorbance spectrum estimation is presented. This method uses multispectral holographic images acquired and reconstructed at a small number (e.g., three to six) of wavelengths, estimates the absorbance spectrum of the sample, and projects it onto a color tristimulus. Using this method, the wavelength selection is optimized to holographically image 25 pathology slide samples with different tissue and stain combinations to significantly reduce color errors in the final reconstructed images. The results can be used as a practical guide for various imaging applications and, in particular, to correct color distortions in holographic imaging of pathology samples spanning different dyes and tissue types.

Deep learning-based super-resolution in coherent imaging systems.

We present a deep learning framework based on a generative adversarial network (GAN) to perform super-resolution in coherent imaging systems. We demonstrate that this framework can enhance the resolution of both pixel size-limited and diffraction-limited coherent imaging systems. The capabilities of this approach are experimentally validated by super-resolving complex-valued images acquired using a lensfree on-chip holographic microscope, the resolution of which was pixel size-limited. Using the same GAN-based approach, we also improved the resolution of a lens-based holographic imaging system that was limited in resolution by the numerical aperture of its objective lens. This deep learning-based super-resolution framework can be broadly applied to enhance the space-bandwidth product of coherent imaging systems using image data and convolutional neural networks, and provides a rapid, non-iterative method for solving inverse image reconstruction or enhancement problems in optics.

Deep learning-based color holographic microscopy.

We report a framework based on a generative adversarial network that performs high-fidelity color image reconstruction using a single hologram of a sample that is illuminated simultaneously by light at three different wavelengths. The trained network learns to eliminate missing-phase-related artifacts, and generates an accurate color transformation for the reconstructed image. Our framework is experimentally demonstrated using lung and prostate tissue sections that are labeled with different histological stains. This framework is envisaged to be applicable to point-of-care histopathology and presents a significant improvement in the throughput of coherent microscopy systems given that only a single hologram of the specimen is required for accurate color imaging.