Intelligent Image-Activated Cell Sorting.

A fundamental challenge of biology is to understand the vast heterogeneity of cells, particularly how cellular composition, structure, and morphology are linked to cellular physiology. Unfortunately, conventional technologies are limited in uncovering these relations. We present a machine-intelligence technology based on a radically different architecture that realizes real-time image-based intelligent cell sorting at an unprecedented rate. This technology, which we refer to as intelligent image-activated cell sorting, integrates high-throughput cell microscopy, focusing, and sorting on a hybrid software-hardware data-management infrastructure, enabling real-time automated operation for data acquisition, data processing, decision-making, and actuation. We use it to demonstrate real-time sorting of microalgal and blood cells based on intracellular protein localization and cell-cell interaction from large heterogeneous populations for studying photosynthesis and atherothrombosis, respectively. The technology is highly versatile and expected to enable machine-based scientific discovery in biological, pharmaceutical, and medical sciences.

High-speed single-pixel imaging by frequency-time-division multiplexing.

We propose and experimentally demonstrate high-speed single-pixel imaging by integrating frequency-division multiplexing and time-division multiplexing (techniques used widely in telecommunications) and applying the combined technique, namely, frequency-time-division multiplexing (FTDM), to optical imaging. Specifically, FTDM single-pixel imaging uses an array of broadband, spatially distributed, dual-frequency combs (i.e., spatial dual combs) for multidimensional illumination and detects an image-encoded time-domain signal with a single-pixel photodetector in a FTDM manner. As a proof-of-principle demonstration, we use the method to show ultrafast two-color (bright-field and fluorescence) single-pixel microscopy of breast cancer cells at a high frame rate of 32,000 fps and ultrafast image velocimetry of fluorescent particles flowing at a high speed of ${ \gt }{2}\;{\rm m/s}$>2m/s.

Simple, stable, compact implementation of frequency-division-multiplexed microscopy by inline interferometry.

Frequency-division-multiplexed (FDM) imaging is a powerful method for high-speed imaging that surpasses the speed limit of conventional imaging constrained by the frame rate of image sensors. However, its complexity, instability, and bulkiness deriving from the implementation with a Mach-Zehnder interferometer hamper its practical applications. Here we demonstrate a simple, stable, and compact implementation of FDM imaging by inline interferometry that makes the method readily available to practical situations. As a proof-of-concept demonstration, we demonstrate 2D bright-field and fluorescence image acquisition of fluorescent beads, microalgal cells, and breast cancer cells within 65.5 µs, corresponding to 15,300 frames per second.

Compact and fully collinear light source for broadband multiplex CARS microscopy covering the fingerprint region.

Compact and fully collinear light source for multiplex coherent anti-Stokes Raman scattering (CARS) microscopy was proposed and demonstrated. It consists of only a microchip laser, a short photonic crystal fiber, and a longpass filter. It offers performance of sensitivity, bandwidth, and spectral resolution suitable for biomedical applications, especially covering the entire fingerprint region (500-1800 cm(-1)). It can be readily implemented by a commercially available microchip laser and a photonic crystal fiber. It has great potential to expand the utility of CARS microscopy to a wide variety of fields such as endoscopy.

Compact light source for ultrabroadband coherent anti-Stoke Raman scattering (CARS) microscopy.

A compact light source module for ultrabroadband coherent anti-Stoke Raman scattering (CARS) microscopy was developed. It mainly consists of a nanosecond microchip laser, a photonic crystal fiber for Stokes light generation, and a single mode polarization maintaining fiber for pump light propagation. It is alignment-free and relatively low-cost compared with previous light sources of CARS microscopy. By using an assembled module, we successfully observed an ultrabroadband CARS spectrum and a CARS image of a murine adipocyte. The module is expected to greatly spread the CARS microscopy to various fields by its extreme easiness to handle.

Quantitative index of arbitrary molar concentration for coherent anti-Stoke Raman scattering (CARS) spectroscopy and microscopy.

We propose a simple quantitative index for coherent anti-Stoke Raman scattering (CARS) spectroscopy and microscopy. Unlike previous similar indices, it can be applied to samples with arbitrary molar concentration, and it is robust against environmental change. Concentrations of aqueous hydrogen peroxide solution and lipid concentration distribution in a live murine adipocyte were successfully quantified by the new index. The index can be obtained in a broad range of CARS setups and it is readily applicable to quantitative CARS microscopy for deep inspection of samples such as biological specimens.

Large-volume focus control at 10 MHz refresh rate via fast line-scanning amplitude-encoded scattering-assisted holography.

The capability of focus control has been central to optical technologies that require both high temporal and spatial resolutions. However, existing varifocal lens schemes are commonly limited to the response time on the microsecond timescale and share the fundamental trade-off between the response time and the tuning power. Here, we propose an ultrafast holographic focusing method enabled by translating the speed of a fast 1D beam scanner into the speed of the complex wavefront modulation of a relatively slow 2D spatial light modulator. Using a pair of a digital micromirror device and a resonant scanner, we demonstrate an unprecedented refresh rate of focus control of 31 MHz, which is more than 1,000 times faster than the switching rate of a digital micromirror device. We also show that multiple micrometer-sized focal spots can be independently addressed in a range of over 1 MHz within a large volume of 5 mm × 5 mm × 5.5 mm, validating the superior spatiotemporal characteristics of the proposed technique - high temporal and spatial precision, high tuning power, and random accessibility in a three-dimensional space. The demonstrated scheme offers a new route towards three-dimensional light manipulation in the 100 MHz regime.

High-throughput fluorescence lifetime imaging flow cytometry.

Flow cytometry is a vital tool in biomedical research and laboratory medicine. However, its accuracy is often compromised by undesired fluctuations in fluorescence intensity. While fluorescence lifetime imaging microscopy (FLIM) bypasses this challenge as fluorescence lifetime remains unaffected by such fluctuations, the full integration of FLIM into flow cytometry has yet to be demonstrated due to speed limitations. Here we overcome the speed limitations in FLIM, thereby enabling high-throughput FLIM flow cytometry at a high rate of over 10,000 cells per second. This is made possible by using dual intensity-modulated continuous-wave beam arrays with complementary modulation frequency pairs for fluorophore excitation and acquiring fluorescence lifetime images of rapidly flowing cells. Moreover, our FLIM system distinguishes subpopulations in male rat glioma and captures dynamic changes in the cell nucleus induced by an anti-cancer drug. FLIM flow cytometry significantly enhances cellular analysis capabilities, providing detailed insights into cellular functions, interactions, and environments.

Virtual-freezing fluorescence imaging flow cytometry with 5-aminolevulinic acid stimulation and antibody labeling for detecting all forms of circulating tumor cells.

Circulating tumor cells (CTCs) are precursors to cancer metastasis. In blood circulation, they take various forms such as single CTCs, CTC clusters, and CTC-leukocyte clusters, all of which have unique characteristics in terms of physiological function and have been a subject of extensive research in the last several years. Unfortunately, conventional methods are limited in accurately analysing the highly heterogeneous nature of CTCs. Here we present an effective strategy for simultaneously analysing all forms of CTCs in blood by virtual-freezing fluorescence imaging (VIFFI) flow cytometry with 5-aminolevulinic acid (5-ALA) stimulation and antibody labeling. VIFFI is an optomechanical imaging method that virtually freezes the motion of fast-flowing cells on an image sensor to enable high-throughput yet sensitive imaging of every single event. 5-ALA stimulates cancer cells to induce the accumulation of protoporphyrin (PpIX), a red fluorescent substance, making it possible to detect all cancer cells even if they show no expression of the epithelial cell adhesion molecule, a typical CTC biomarker. Although PpIX signals are generally weak, VIFFI flow cytometry can detect them by virtue of its high sensitivity. As a proof-of-principle demonstration of the strategy, we applied cancer cells spiked in blood to the strategy to demonstrate image-based detection and accurate classification of single cancer cells, clusters of cancer cells, and clusters of a cancer cell(s) and a leukocyte(s). To show the clinical utility of our method, we used it to evaluate blood samples of four breast cancer patients and four healthy donors and identified EpCAM-positive PpIX-positive cells in one of the patient samples. Our work paves the way toward the determination of cancer prognosis, the guidance and monitoring of treatment, and the design of antitumor strategies for cancer patients.

Deep imaging flow cytometry.

Imaging flow cytometry (IFC) has become a powerful tool for diverse biomedical applications by virtue of its ability to image single cells in a high-throughput manner. However, there remains a challenge posed by the fundamental trade-off between throughput, sensitivity, and spatial resolution. Here we present deep-learning-enhanced imaging flow cytometry (dIFC) that circumvents this trade-off by implementing an image restoration algorithm on a virtual-freezing fluorescence imaging (VIFFI) flow cytometry platform, enabling higher throughput without sacrificing sensitivity and spatial resolution. A key component of dIFC is a high-resolution (HR) image generator that synthesizes "virtual" HR images from the corresponding low-resolution (LR) images acquired with a low-magnification lens (10×/0.4-NA). For IFC, a low-magnification lens is favorable because of reduced image blur of cells flowing at a higher speed, which allows higher throughput. We trained and developed the HR image generator with an architecture containing two generative adversarial networks (GANs). Furthermore, we developed dIFC as a method by combining the trained generator and IFC. We characterized dIFC using Chlamydomonas reinhardtii cell images, fluorescence in situ hybridization (FISH) images of Jurkat cells, and Saccharomyces cerevisiae (budding yeast) cell images, showing high similarities of dIFC images to images obtained with a high-magnification lens (40×/0.95-NA), at a high flow speed of 2 m s-1. We lastly employed dIFC to show enhancements in the accuracy of FISH-spot counting and neck-width measurement of budding yeast cells. These results pave the way for statistical analysis of cells with high-dimensional spatial information.

AI on a chip.

Artificial intelligence (AI) has dramatically changed the landscape of science, industry, defence, and medicine in the last several years. Supported by considerably enhanced computational power and cloud storage, the field of AI has shifted from mostly theoretical studies in the discipline of computer science to diverse real-life applications such as drug design, material discovery, speech recognition, self-driving cars, advertising, finance, medical imaging, and astronomical observation, where AI-produced outcomes have been proven to be comparable or even superior to the performance of human experts. In these applications, what is essentially important for the development of AI is the data needed for machine learning. Despite its prominent importance, the very first process of the AI development, namely data collection and data preparation, is typically the most laborious task and is often a limiting factor of constructing functional AI algorithms. Lab-on-a-chip technology, in particular microfluidics, is a powerful platform for both the construction and implementation of AI in a large-scale, cost-effective, high-throughput, automated, and multiplexed manner, thereby overcoming the above bottleneck. On this platform, high-throughput imaging is a critical tool as it can generate high-content information (e.g., size, shape, structure, composition, interaction) of objects on a large scale. High-throughput imaging can also be paired with sorting and DNA/RNA sequencing to conduct a massive survey of phenotype-genotype relations whose data is too complex to analyze with traditional computational tools, but is analyzable with the power of AI. In addition to its function as a data provider, lab-on-a-chip technology can also be employed to implement the developed AI for accurate identification, characterization, classification, and prediction of objects in mixed, heterogeneous, or unknown samples. In this review article, motivated by the excellent synergy between AI and lab-on-a-chip technology, we outline fundamental elements, recent advances, future challenges, and emerging opportunities of AI with lab-on-a-chip technology or "AI on a chip" for short.

Virtual-freezing fluorescence imaging flow cytometry.

By virtue of the combined merits of flow cytometry and fluorescence microscopy, imaging flow cytometry (IFC) has become an established tool for cell analysis in diverse biomedical fields such as cancer biology, microbiology, immunology, hematology, and stem cell biology. However, the performance and utility of IFC are severely limited by the fundamental trade-off between throughput, sensitivity, and spatial resolution. Here we present an optomechanical imaging method that overcomes the trade-off by virtually freezing the motion of flowing cells on the image sensor to effectively achieve 1000 times longer exposure time for microscopy-grade fluorescence image acquisition. Consequently, it enables high-throughput IFC of single cells at >10,000 cells s-1 without sacrificing sensitivity and spatial resolution. The availability of numerous information-rich fluorescence cell images allows high-dimensional statistical analysis and accurate classification with deep learning, as evidenced by our demonstration of unique applications in hematology and microbiology.

Raman image-activated cell sorting.

The advent of image-activated cell sorting and imaging-based cell picking has advanced our knowledge and exploitation of biological systems in the last decade. Unfortunately, they generally rely on fluorescent labeling for cellular phenotyping, an indirect measure of the molecular landscape in the cell, which has critical limitations. Here we demonstrate Raman image-activated cell sorting by directly probing chemically specific intracellular molecular vibrations via ultrafast multicolor stimulated Raman scattering (SRS) microscopy for cellular phenotyping. Specifically, the technology enables real-time SRS-image-based sorting of single live cells with a throughput of up to ~100 events per second without the need for fluorescent labeling. To show the broad utility of the technology, we show its applicability to diverse cell types and sizes. The technology is highly versatile and holds promise for numerous applications that are previously difficult or undesirable with fluorescence-based technologies.

Intelligent image-activated cell sorting 2.0.

The advent of intelligent image-activated cell sorting (iIACS) has enabled high-throughput intelligent image-based sorting of single live cells from heterogeneous populations. iIACS is an on-chip microfluidic technology that builds on a seamless integration of a high-throughput fluorescence microscope, cell focuser, cell sorter, and deep neural network on a hybrid software-hardware data management architecture, thereby providing the combined merits of optical microscopy, fluorescence-activated cell sorting (FACS), and deep learning. Here we report an iIACS machine that far surpasses the state-of-the-art iIACS machine in system performance in order to expand the range of applications and discoveries enabled by the technology. Specifically, it provides a high throughput of ∼2000 events per second and a high sensitivity of ∼50 molecules of equivalent soluble fluorophores (MESFs), both of which are 20 times superior to those achieved in previous reports. This is made possible by employing (i) an image-sensor-based optomechanical flow imaging method known as virtual-freezing fluorescence imaging and (ii) a real-time intelligent image processor on an 8-PC server equipped with 8 multi-core CPUs and GPUs for intelligent decision-making, in order to significantly boost the imaging performance and computational power of the iIACS machine. We characterize the iIACS machine with fluorescent particles and various cell types and show that the performance of the iIACS machine is close to its achievable design specification. Equipped with the improved capabilities, this new generation of the iIACS technology holds promise for diverse applications in immunology, microbiology, stem cell biology, cancer biology, pathology, and synthetic biology.

A practical guide to intelligent image-activated cell sorting.

Intelligent image-activated cell sorting (iIACS) is a machine-intelligence technology that performs real-time intelligent image-based sorting of single cells with high throughput. iIACS extends beyond the capabilities of fluorescence-activated cell sorting (FACS) from fluorescence intensity profiles of cells to multidimensional images, thereby enabling high-content sorting of cells or cell clusters with unique spatial chemical and morphological traits. Therefore, iIACS serves as an integral part of holistic single-cell analysis by enabling direct links between population-level analysis (flow cytometry), cell-level analysis (microscopy), and gene-level analysis (sequencing). Specifically, iIACS is based on a seamless integration of high-throughput cell microscopy (e.g., multicolor fluorescence imaging, bright-field imaging), cell focusing, cell sorting, and deep learning on a hybrid software-hardware data management infrastructure, enabling real-time automated operation for data acquisition, data processing, intelligent decision making, and actuation. Here, we provide a practical guide to iIACS that describes how to design, build, characterize, and use an iIACS machine. The guide includes the consideration of several important design parameters, such as throughput, sensitivity, dynamic range, image quality, sort purity, and sort yield; the development and integration of optical, microfluidic, electrical, computational, and mechanical components; and the characterization and practical usage of the integrated system. Assuming that all components are readily available, a team of several researchers experienced in optics, electronics, digital signal processing, microfluidics, mechatronics, and flow cytometry can complete this protocol in ~3 months.

Author Correction: A practical guide to intelligent image-activated cell sorting.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

On-chip light-sheet fluorescence imaging flow cytometry at a high flow speed of 1 m/s.

We present on-chip fluorescence imaging flow cytometry by light-sheet excitation on a mirror-embedded microfluidic chip. The method allows us to obtain microscopy-grade fluorescence images of cells flowing at a high speed of 1 m/s, which is comparable to the flow speed of conventional non-imaging flow cytometers. To implement the light-sheet excitation of flowing cells in a microchannel, we designed and fabricated a mirror-embedded PDMS-based microfluidic chip. To show its broad utility, we used the method to classify large populations of microalgal cells (Euglena gracilis) and human cancer cells (human adenocarcinoma cells). Our method holds promise for large-scale single-cell analysis.

High-throughput imaging flow cytometry by optofluidic time-stretch microscopy.

The ability to rapidly assay morphological and intracellular molecular variations within large heterogeneous populations of cells is essential for understanding and exploiting cellular heterogeneity. Optofluidic time-stretch microscopy is a powerful method for meeting this goal, as it enables high-throughput imaging flow cytometry for large-scale single-cell analysis of various cell types ranging from human blood to algae, enabling a unique class of biological, medical, pharmaceutical, and green energy applications. Here, we describe how to perform high-throughput imaging flow cytometry by optofluidic time-stretch microscopy. Specifically, this protocol provides step-by-step instructions on how to build an optical time-stretch microscope and a cell-focusing microfluidic device for optofluidic time-stretch microscopy, use it for high-throughput single-cell image acquisition with sub-micrometer resolution at >10,000 cells per s, conduct image construction and enhancement, perform image analysis for large-scale single-cell analysis, and use computational tools such as compressive sensing and machine learning for handling the cellular 'big data'. Assuming all components are readily available, a research team of three to four members with an intermediate level of experience with optics, electronics, microfluidics, digital signal processing, and sample preparation can complete this protocol in a time frame of 1 month.

Rapid mesoscale multiphoton microscopy of human skin.

We present a multiphoton microscope designed for mesoscale imaging of human skin. The system is based on two-photon excited fluorescence and second-harmonic generation, and images areas of ~0.8x0.8 mm2 at speeds of 0.8 fps (800x800 pixels; 12 frame averages) for high signal-to-noise ratio, with lateral and axial resolutions of 0.5µm and 3.3µm, respectively. The main novelty of this instrument is the design of the scan head, which includes a fast galvanometric scanner, optimized relay optics, a beam expander and high NA objective lens. Computed aberrations in focus are below the Marechal criterion of 0.07λ rms for diffraction-limited performance. We demonstrate the practical utility of this microscope by ex-vivo imaging of wide areas in normal human skin.

New high-efficiency source of a three-photon w state and its full characterization using quantum state tomography.

We propose and demonstrate a new scheme of high-efficiency generation of the three-photon polarization-entangled W state, which is a typical three-qubit entangled state. The high efficiency has enabled the first full characterization of the state by quantum state tomography. We have analyzed the obtained state and observed its nature of tripartite entanglement and robustness of entanglement.