Artificial intelligence-enabled quantitative phase imaging methods for life sciences.

Quantitative phase imaging, integrated with artificial intelligence, allows for the rapid and label-free investigation of the physiology and pathology of biological systems. This review presents the principles of various two-dimensional and three-dimensional label-free phase imaging techniques that exploit refractive index as an intrinsic optical imaging contrast. In particular, we discuss artificial intelligence-based analysis methodologies for biomedical studies including image enhancement, segmentation of cellular or subcellular structures, classification of types of biological samples and image translation to furnish subcellular and histochemical information from label-free phase images. We also discuss the advantages and challenges of artificial intelligence-enabled quantitative phase imaging analyses, summarize recent notable applications in the life sciences, and cover the potential of this field for basic and industrial research in the life sciences.

Inference in artificial intelligence with deep optics and photonics.

Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.

Virtual Staining of Nonfixed Tissue Histology.

Surgical pathology workflow involves multiple labor-intensive steps, such as tissue removal, fixation, embedding, sectioning, staining, and microscopic examination. This process is time-consuming and costly and requires skilled technicians. In certain clinical scenarios, such as intraoperative consultations, there is a need for faster histologic evaluation to provide real-time surgical guidance. Currently, frozen section techniques involving hematoxylin and eosin (H&E) staining are used for intraoperative pathology consultations. However, these techniques have limitations, including a turnaround time of 20 to 30 minutes, staining artifacts, and potential tissue loss, negatively impacting accurate diagnosis. To address these challenges, researchers are exploring alternative optical imaging modalities for rapid microscopic tissue imaging. These modalities differ in optical characteristics, tissue preparation requirements, imaging equipment, and output image quality and format. Some of these imaging methods have been combined with computational algorithms to generate H&E-like images, which could greatly facilitate their adoption by pathologists. Here, we provide a comprehensive, organ-specific review of the latest advancements in emerging imaging modalities applied to nonfixed human tissue. We focused on studies that generated H&E-like images evaluated by pathologists. By presenting up-to-date research progress and clinical utility, this review serves as a valuable resource for scholars and clinicians, covering some of the major technical developments in this rapidly evolving field. It also offers insights into the potential benefits and drawbacks of alternative imaging modalities and their implications for improving patient care.

Deep Learning-Enabled Multiplexed Point-of-Care Sensor using a Paper-Based Fluorescence Vertical Flow Assay.

Multiplexed computational sensing with a point-of-care serodiagnosis assay to simultaneously quantify three biomarkers of acute cardiac injury is demonstrated. This point-of-care sensor includes a paper-based fluorescence vertical flow assay (fxVFA) processed by a low-cost mobile reader, which quantifies the target biomarkers through trained neural networks, all within <15 min of test time using 50 µL of serum sample per patient. This fxVFA platform is validated using human serum samples to quantify three cardiac biomarkers, i.e., myoglobin, creatine kinase-MB, and heart-type fatty acid binding protein, achieving less than 0.52 ng mL-1 limit-of-detection for all three biomarkers with minimal cross-reactivity. Biomarker concentration quantification using the fxVFA that is coupled to neural network-based inference is blindly tested using 46 individually activated cartridges, which shows a high correlation with the ground truth concentrations for all three biomarkers achieving >0.9 linearity and <15% coefficient of variation. The competitive performance of this multiplexed computational fxVFA along with its inexpensive paper-based design and handheld footprint makes it a promising point-of-care sensor platform that can expand access to diagnostics in resource-limited settings.

Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.

We demonstrate that a deep neural network can be trained to virtually refocus a two-dimensional fluorescence image onto user-defined three-dimensional (3D) surfaces within the sample. Using this method, termed Deep-Z, we imaged the neuronal activity of a Caenorhabditis elegans worm in 3D using a time sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field by 20-fold without any axial scanning, additional hardware or a trade-off of imaging resolution and speed. Furthermore, we demonstrate that this approach can correct for sample drift, tilt and other aberrations, all digitally performed after the acquisition of a single fluorescence image. This framework also cross-connects different imaging modalities to each other, enabling 3D refocusing of a single wide-field fluorescence image to match confocal microscopy images acquired at different sample planes. Deep-Z has the potential to improve volumetric imaging speed while reducing challenges relating to sample drift, aberration and defocusing that are associated with standard 3D fluorescence microscopy.

Deep learning enables cross-modality super-resolution in fluorescence microscopy.

We present deep-learning-enabled super-resolution across different fluorescence microscopy modalities. This data-driven approach does not require numerical modeling of the imaging process or the estimation of a point-spread-function, and is based on training a generative adversarial network (GAN) to transform diffraction-limited input images into super-resolved ones. Using this framework, we improve the resolution of wide-field images acquired with low-numerical-aperture objectives, matching the resolution that is acquired using high-numerical-aperture objectives. We also demonstrate cross-modality super-resolution, transforming confocal microscopy images to match the resolution acquired with a stimulated emission depletion (STED) microscope. We further demonstrate that total internal reflection fluorescence (TIRF) microscopy images of subcellular structures within cells and tissues can be transformed to match the results obtained with a TIRF-based structured illumination microscope. The deep network rapidly outputs these super-resolved images, without any iterations or parameter search, and could serve to democratize super-resolution imaging.

Monitoring gestational diabetes at the point-of-care via dual glycated albumin lateral flow assays in conjunction with a handheld reader.

Chronic conditions like diabetes require monitoring of vital biomarkers over extended periods of time. Monitoring gestational diabetes mellitus (GDM) is crucial to avoid short- and long-term adverse effects on both mother and infant. Providing monitoring systems to patients at the point-of-care (POC) has the potential to help mitigate these effects. In this manuscript, we propose the use of a sensing system combining lateral flow assays (LFAs) with a handheld colorimetric reader for use in tracking the glycemic status of a GDM patient at the POC. Current strategies of GDM monitoring include glucose and HbA1c measurements. These are often too frequent or not frequent enough for effective monitoring. Hence, we have developed a sensor for an intermediate interval biomarker - glycated albumin (GA). Based on the half-life of the protein, GA is measured once every 2-3 weeks. Here we first present two lateral flow assays, one for GA and another for total serum albumin used in conjunction with a handheld reader to read the colorimetric signals. Both assays have a sandwich aptamer format and measure the target proteins in their physiologically relevant ranges. The GA assay has a dynamic range of 3-20 mg ml-1 and the serum albumin assay has a range of 20-50 mg ml-1 without any sample dilution. Both LFAs were then incorporated into a single dual assay cartridge such that both assays could run simultaneously and provide the % glycated albumin value from a single test. Thus, the dual assay cartridge plus reader system has the potential to provide an effective platform for measuring GA for tracking GDM at the POC.

Roadmap on digital holography [Invited].

This Roadmap article on digital holography provides an overview of a vast array of research activities in the field of digital holography. The paper consists of a series of 25 sections from the prominent experts in digital holography presenting various aspects of the field on sensing, 3D imaging and displays, virtual and augmented reality, microscopy, cell identification, tomography, label-free live cell imaging, and other applications. Each section represents the vision of its author to describe the significant progress, potential impact, important developments, and challenging issues in the field of digital holography.

Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks.

Optical machine learning offers advantages in terms of power efficiency, scalability and computation speed. Recently, an optical machine learning method based on Diffractive Deep Neural Networks (D2NNs) has been introduced to execute a function as the input light diffracts through passive layers, designed by deep learning using a computer. Here we introduce improvements to D2NNs by changing the training loss function and reducing the impact of vanishing gradients in the error back-propagation step. Using five phase-only diffractive layers, we numerically achieved a classification accuracy of 97.18% and 89.13% for optical recognition of handwritten digits and fashion products, respectively; using both phase and amplitude modulation (complex-valued) at each layer, our inference performance improved to 97.81% and 89.32%, respectively. Furthermore, we report the integration of D2NNs with electronic neural networks to create hybrid-classifiers that significantly reduce the number of input pixels into an electronic network using an ultra-compact front-end D2NN with a layer-to-layer distance of a few wavelengths, also reducing the complexity of the successive electronic network. Using a 5-layer phase-only D2NN jointly-optimized with a single fully-connected electronic layer, we achieved a classification accuracy of 98.71% and 90.04% for the recognition of handwritten digits and fashion products, respectively. Moreover, the input to the electronic network was compressed by >7.8 times down to 10×10 pixels. Beyond creating low-power and high-frame rate machine learning platforms, D2NN-based hybrid neural networks will find applications in smart optical imager and sensor design.

Measurement of serum phosphate levels using a mobile sensor.

The measurement of serum phosphate concentration is crucial for patients with advanced chronic kidney disease (CKD) and those on maintenance dialysis, as abnormal phosphate levels may be associated with severe health risks. It is important to monitor serum phosphate levels on a regular basis in these patients; however, such measurements are generally limited to every 0.5-3 months, depending on the severity of CKD. This is due to the fact that serum phosphate measurements can only be performed at regular clinic visits, in addition to cost considerations. Here we present a portable and cost-effective point-of-care device capable of measuring serum phosphate levels using a single drop of blood (<60 µl). This is achieved by integrating a paper-based microfluidic platform with a custom-designed smartphone reader. This mobile sensor was tested on patients undergoing dialysis, where whole blood samples were acquired before starting the hemodialysis and during the three-hour treatment. This sampling during the hemodialysis, under patient consent, allowed us to test blood samples with a wide range of phosphate concentrations, and our results showed a strong correlation with the ground truth laboratory tests performed on the same patient samples (Pearson coefficient r = 0.95 and p < 0.001). Our 3D-printed smartphone attachment weighs about 400 g and costs less than 80 USD, whereas the material cost for the disposable test is <3.5 USD (under low volume manufacturing). This low-cost and easy-to-operate system can be used to measure serum phosphate levels at the point-of-care in about 45 min and can potentially be used on a daily basis by patients at home.

Inkjet-printed point-of-care immunoassay on a nanoscale polymer brush enables subpicomolar detection of analytes in blood.

The ELISA is the mainstay for sensitive and quantitative detection of protein analytes. Despite its utility, ELISA is time-consuming, resource-intensive, and infrastructure-dependent, limiting its availability in resource-limited regions. Here, we describe a self-contained immunoassay platform (the "D4 assay") that converts the sandwich immunoassay into a point-of-care test (POCT). The D4 assay is fabricated by inkjet printing assay reagents as microarrays on nanoscale polymer brushes on glass chips, so that all reagents are "on-chip," and these chips show durable storage stability without cold storage. The D4 assay can interrogate multiple analytes from a drop of blood, is compatible with a smartphone detector, and displays analytical figures of merit that are comparable to standard laboratory-based ELISA in whole blood. These attributes of the D4 POCT have the potential to democratize access to high-performance immunoassays in resource-limited settings without sacrificing their performance.

Smartphone-imaged microfluidic biochip for measuring CD64 expression from whole blood.

Sepsis, a life-threatening syndrome that contributes to millions of deaths annually worldwide, represents a moral and economic burden to the healthcare system. Although no single, or even a combination of biomarkers has been validated for the diagnosis of sepsis, multiple studies have shown the high specificity of CD64 expression on neutrophils (nCD64) to sepsis. The analysis of elevated nCD64 in the first 2-6 hours after infection during the pro-inflammatory stage could significantly contribute to early sepsis diagnosis. Therefore, a rapid and automated device to periodically measure nCD64 expression at the point-of-care (POC) could lead to timely medical intervention and reduced mortality rates. Current accepted technologies for measuring nCD64 expression, such as flow cytometry, require manual sample preparation and long incubation times. For POC applications, however, the technology should be able to measure nCD64 expression with little to no sample preparation. In this paper, we demonstrate a smartphone-imaged microfluidic biochip for detecting nCD64 expression in under 50 min. In our assay, first unprocessed whole blood is injected into a capture chamber to immunologically capture nCD64 along a staggered array of pillars, which were previously functionalized with an antibody against CD64. Then, an image of the capture channel is taken using a smartphone-based microscope. This image is used to measure the cumulative fraction of captured cells (γ) as a function of length in the channel. During the image analysis, a statistical model is fitted to γ in order to extract the probability of capture of neutrophils per collision with a pillar (ε). The fitting shows a strong correlation with nCD64 expression measured using flow cytometry (R2 = 0.82). Finally, the applicability of the device to sepsis was demonstrated by analyzing nCD64 from 8 patients (37 blood samples analyzed) along the time they were admitted to the hospital. Results from this analysis, obtained using the smartphone-imaged microfluidic biochip were compared with flow cytometry. Again, a correlation coefficient R2 = 0.82 (slope = 0.99) was obtained demonstrating a good linear correlation between the two techniques. Deployment of this technology in ICU could significantly enhance patient care worldwide.

Lensless digital holographic microscopy and its applications in biomedicine and environmental monitoring.

Optical compound microscope has been a major tool in biomedical imaging for centuries. Its performance relies on relatively complicated, bulky and expensive lenses and alignment mechanics. In contrast, the lensless microscope digitally reconstructs microscopic images of specimens without using any lenses, as a result of which it can be made much smaller, lighter and lower-cost. Furthermore, the limited space-bandwidth product of objective lenses in a conventional microscope can be significantly surpassed by a lensless microscope. Such lensless imaging designs have enabled high-resolution and high-throughput imaging of specimens using compact, portable and cost-effective devices to potentially address various point-of-care, global-health and telemedicine related challenges. In this review, we discuss the operation principles and the methods behind lensless digital holographic on-chip microscopy. We also go over various applications that are enabled by cost-effective and compact implementations of lensless microscopy, including some recent work on air quality monitoring, which utilized machine learning for high-throughput and accurate quantification of particulate matter in air. Finally, we conclude with a brief future outlook of this computational imaging technology.

Ferrodrop Dose-Optimized Digital Quantification of Biomolecules in Low-Volume Samples.

We present an approach to estimate the concentration of a biomolecule in a solution by sampling several nanoliter-scale volumes and determining if the volumes contain any biomolecules. In this method, varying volume fractions (nanoliter-scale) of a sample of nucleic acids are introduced to an array of uniform volume reaction wells (100 µL), which are then fluorescently imaged to determine if signal is above a threshold after nucleic acid amplification, all without complex instrumentation. The nanoliter volumes are generated and introduced using the simple positioning of a permanent magnet, and imaging is performed with a cellphone-based fluorescence detection scheme, both methods suitable for limited-resource settings. We use the length of time a magnetic field is applied to generate a calibrated number of nanoliter ferrodrops of sample mixed with ferrofluid at a step emulsification microfluidic junction. Each dose of ferrodrops is then transferred into larger microliter scale reaction wells on chip through a simple shift of the external magnet. Nucleic acid amplification is achieved using loop-mediated isothermal amplification (LAMP). By repeating each nanoliter dosage a number of times to calculate the probability of a positive signal at each dosage, we can use a binomial probability distribution to estimate the sample nucleic acid concentration. Using this approach we demonstrate detection of lambda DNA molecules down to 25 copies per microliter. The ability to dose separate nanoliter-scale volumes of a low-volume sample across wells in this platform is suited for multiplexed assays. This platform has the potential to be applied to a range of diseases by mixing a sample with magnetic nanoparticles.

Enzyme-Free Nucleic Acid Amplification Assay Using a Cellphone-Based Well Plate Fluorescence Reader.

Nucleic acids, DNA and RNA, provide important fingerprint information for various pathogens and have significant diagnostic value; however, improved approaches are urgently needed to enable rapid detection of nucleic acids in simple point-of-care formats with high sensitivity and specificity. Here, we present a system that utilizes a series of toehold-triggered hybridization/displacement reactions that are designed to convert a given amount of RNA molecules (i.e., the analyte) into an amplified amount of signaling molecules without any washing steps or thermocycling. Fluorescent probes for signal generation were designed to consume products of the catalytic reaction in order to push the equilibrium and enhance the assay fold amplification for improved sensitivity and reaction speed. The system of toehold-assisted reactions is also modeled to better understand its performance and capabilities, and we empirically demonstrate the success of this approach with two analytes of diagnostic importance, i.e., influenza viral RNA and a micro RNA (miR-31). We also show that the amplified signal permits using a compact and cost-effective smartphone-based fluorescence reader, an important requirement toward a nucleic-acid-based point-of-care diagnostic system.

Optical refractometry using lensless holography and autofocusing.

Conventional optical refractometry methods are often limited by a narrow measurement range, complex hardware, or relatively high cost. Here, we present a novel refractometry method to measure the bulk refractive index (RI) of materials (including solids and liquids) using lensless holographic on-chip imaging and autofocusing, which is simple, cost-effective, and has a large RI measurement range. As a proof of concept, two compact prototypes were built to measure the RIs of solid materials and liquids, respectively, and they were tested by measuring the RIs of a ZnSe plate and a microscopy immersion oil. Experimental results show that our devices have an average accuracy of ~3 × 10-4 RI unit (RIU) with an estimated precision of ~3 × 10-3 RIU for solids; and an average accuracy of ~1 × 10-4 RIU with an estimated precision of ~3 × 10-4 RIU for liquids. We believe that this cost-effective and portable RI measurement platform holds promise to be used in laboratory and industrial settings.

Computational imaging of sperm locomotion.

Not only essential for scientific research, but also in the analysis of male fertility and for animal husbandry, sperm tracking and characterization techniques have been greatly benefiting from computational imaging. Digital image sensors, in combination with optical microscopy tools and powerful computers, have enabled the use of advanced detection and tracking algorithms that automatically map sperm trajectories and calculate various motility parameters across large data sets. Computational techniques are driving the field even further, facilitating the development of unconventional sperm imaging and tracking methods that do not rely on standard optical microscopes and objective lenses, which limit the field of view and volume of the semen sample that can be imaged. As an example, a holographic on-chip sperm imaging platform, only composed of a light-emitting diode and an opto-electronic image sensor, has emerged as a high-throughput, low-cost and portable alternative to lens-based traditional sperm imaging and tracking methods. In this approach, the sample is placed very close to the image sensor chip, which captures lensfree holograms generated by the interference of the background illumination with the light scattered from sperm cells. These holographic patterns are then digitally processed to extract both the amplitude and phase information of the spermatozoa, effectively replacing the microscope objective lens with computation. This platform has further enabled high-throughput 3D imaging of spermatozoa with submicron 3D positioning accuracy in large sample volumes, revealing various rare locomotion patterns. We believe that computational chip-scale sperm imaging and 3D tracking techniques will find numerous opportunities in both sperm related research and commercial applications.

Lensless Imaging and Sensing.

High-resolution optical microscopy has traditionally relied on high-magnification and high-numerical aperture objective lenses. In contrast, lensless microscopy can provide high-resolution images without the use of any focusing lenses, offering the advantages of a large field of view, high resolution, cost-effectiveness, portability, and depth-resolved three-dimensional (3D) imaging. Here we review various approaches to lensless imaging, as well as its applications in biosensing, diagnostics, and cytometry. These approaches include shadow imaging, fluorescence, holography, superresolution 3D imaging, iterative phase recovery, and color imaging. These approaches share a reliance on computational techniques, which are typically necessary to reconstruct meaningful images from the raw data captured by digital image sensors. When these approaches are combined with physical innovations in sample preparation and fabrication, lensless imaging can be used to image and sense cells, viruses, nanoparticles, and biomolecules. We conclude by discussing several ways in which lensless imaging and sensing might develop in the near future.

Edge sparsity criterion for robust holographic autofocusing.

Autofocusing is essential to digital holographic imaging. Previously used autofocusing criteria exhibit challenges when applied to, e.g., connected objects with different optical properties. Furthermore, in some of the earlier autofocusing criteria, the polarity, i.e., whether to search for the peak or the valley as a function of depth, changes for different types of samples, which creates another challenge. Here, we propose a robust and accurate autofocusing criterion that is based on the edge sparsity of the complex optical wavefront, which we termed the "sparsity of the gradient" (SoG). We demonstrated the success of SoG by imaging a wide range of objects, including resolution test targets, stained and unstained Papanicolaou smears, stained tissue sections, and blood smears.

Unconventional methods of imaging: computational microscopy and compact implementations.

In the past two decades or so, there has been a renaissance of optical microscopy research and development. Much work has been done in an effort to improve the resolution and sensitivity of microscopes, while at the same time to introduce new imaging modalities, and make existing imaging systems more efficient and more accessible. In this review, we look at two particular aspects of this renaissance: computational imaging techniques and compact imaging platforms. In many cases, these aspects go hand-in-hand because the use of computational techniques can simplify the demands placed on optical hardware in obtaining a desired imaging performance. In the first main section, we cover lens-based computational imaging, in particular, light-field microscopy, structured illumination, synthetic aperture, Fourier ptychography, and compressive imaging. In the second main section, we review lensfree holographic on-chip imaging, including how images are reconstructed, phase recovery techniques, and integration with smart substrates for more advanced imaging tasks. In the third main section we describe how these and other microscopy modalities have been implemented in compact and field-portable devices, often based around smartphones. Finally, we conclude with some comments about opportunities and demand for better results, and where we believe the field is heading.