Neuromorphic computing for content-based image retrieval.

Neuromorphic computing mimics the neural activity of the brain through emulating spiking neural networks. In numerous machine learning tasks, neuromorphic chips are expected to provide superior solutions in terms of cost and power efficiency. Here, we explore the application of Loihi, a neuromorphic computing chip developed by Intel, for the computer vision task of image retrieval. We evaluated the functionalities and the performance metrics that are critical in content-based visual search and recommender systems using deep-learning embeddings. Our results show that the neuromorphic solution is about 2.5 times more energy-efficient compared with an ARM Cortex-A72 CPU and 12.5 times more energy-efficient compared with NVIDIA T4 GPU for inference by a lightweight convolutional neural network when batch size is 1 while maintaining the same level of matching accuracy. The study validates the potential of neuromorphic computing in low-power image retrieval, as a complementary paradigm to the existing von Neumann architectures.

Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry.

Deep learning has achieved spectacular performance in image and speech recognition and synthesis. It outperforms other machine learning algorithms in problems where large amounts of data are available. In the area of measurement technology, instruments based on the photonic time stretch have established record real-time measurement throughput in spectroscopy, optical coherence tomography, and imaging flow cytometry. These extreme-throughput instruments generate approximately 1 Tbit/s of continuous measurement data and have led to the discovery of rare phenomena in nonlinear and complex systems as well as new types of biomedical instruments. Owing to the abundance of data they generate, time-stretch instruments are a natural fit to deep learning classification. Previously we had shown that high-throughput label-free cell classification with high accuracy can be achieved through a combination of time-stretch microscopy, image processing and feature extraction, followed by deep learning for finding cancer cells in the blood. Such a technology holds promise for early detection of primary cancer or metastasis. Here we describe a new deep learning pipeline, which entirely avoids the slow and computationally costly signal processing and feature extraction steps by a convolutional neural network that directly operates on the measured signals. The improvement in computational efficiency enables low-latency inference and makes this pipeline suitable for cell sorting via deep learning. Our neural network takes less than a few milliseconds to classify the cells, fast enough to provide a decision to a cell sorter for real-time separation of individual target cells. We demonstrate the applicability of our new method in the classification of OT-II white blood cells and SW-480 epithelial cancer cells with more than 95% accuracy in a label-free fashion.

Matrix Analysis of Warped Stretch Imaging.

Sensitive and fast optical imaging is needed for scientific instruments, machine vision, and biomedical diagnostics. Many of the fundamental challenges are addressed with time stretch imaging, which has been used for ultrafast continuous imaging for a diverse range of applications, such as biomarker-free cell classification, the monitoring of laser ablation, and the inspection of flat panel displays. With frame rates exceeding a million scans per second, the firehose of data generated by the time stretch camera requires optical data compression. Warped stretch imaging technology utilizes nonuniform spectrotemporal optical operations to compress the image in a single-shot real-time fashion. Here, we present a matrix analysis method for the evaluation of these systems and quantify important design parameters and the spatial resolution. The key principles of the system include (1) time/warped stretch transformation and (2) the spatial dispersion of ultrashort optical pulse, which are traced with simple computation of ray-pulse matrix. Furthermore, a mathematical model is constructed for the simulation of imaging operations while considering the optical and electrical response of the system. The proposed analysis method was applied to an example time stretch imaging system via simulation and validated with experimental data.

Context-Aware Image Compression.

We describe a physics-based data compression method inspired by the photonic time stretch wherein information-rich portions of the data are dilated in a process that emulates the effect of group velocity dispersion on temporal signals. With this coding operation, the data can be downsampled at a lower rate than without it. In contrast to previous implementation of the warped stretch compression, here the decoding can be performed without the need of phase recovery. We present rate-distortion analysis and show improvement in PSNR compared to compression via uniform downsampling.

Deep Learning in Label-free Cell Classification.

Label-free cell analysis is essential to personalized genomics, cancer diagnostics, and drug development as it avoids adverse effects of staining reagents on cellular viability and cell signaling. However, currently available label-free cell assays mostly rely only on a single feature and lack sufficient differentiation. Also, the sample size analyzed by these assays is limited due to their low throughput. Here, we integrate feature extraction and deep learning with high-throughput quantitative imaging enabled by photonic time stretch, achieving record high accuracy in label-free cell classification. Our system captures quantitative optical phase and intensity images and extracts multiple biophysical features of individual cells. These biophysical measurements form a hyperdimensional feature space in which supervised learning is performed for cell classification. We compare various learning algorithms including artificial neural network, support vector machine, logistic regression, and a novel deep learning pipeline, which adopts global optimization of receiver operating characteristics. As a validation of the enhanced sensitivity and specificity of our system, we show classification of white blood T-cells against colon cancer cells, as well as lipid accumulating algal strains for biofuel production. This system opens up a new path to data-driven phenotypic diagnosis and better understanding of the heterogeneous gene expressions in cells.

Design of Warped Stretch Transform.

Time stretch dispersive Fourier transform enables real-time spectroscopy at the repetition rate of million scans per second. High-speed real-time instruments ranging from analog-to-digital converters to cameras and single-shot rare-phenomena capture equipment with record performance have been empowered by it. Its warped stretch variant, realized with nonlinear group delay dispersion, offers variable-rate spectral domain sampling, as well as the ability to engineer the time-bandwidth product of the signal's envelope to match that of the data acquisition systems. To be able to reconstruct the signal with low loss, the spectrotemporal distribution of the signal spectrum needs to be sparse. Here, for the first time, we show how to design the kernel of the transform and specifically, the nonlinear group delay profile dictated by the signal sparsity. Such a kernel leads to smart stretching with nonuniform spectral resolution, having direct utility in improvement of data acquisition rate, real-time data compression, and enhancement of ultrafast data capture accuracy. We also discuss the application of warped stretch transform in spectrotemporal analysis of continuous-time signals.

Optical data compression in time stretch imaging.

Time stretch imaging offers real-time image acquisition at millions of frames per second and subnanosecond shutter speed, and has enabled detection of rare cancer cells in blood with record throughput and specificity. An unintended consequence of high throughput image acquisition is the massive amount of digital data generated by the instrument. Here we report the first experimental demonstration of real-time optical image compression applied to time stretch imaging. By exploiting the sparsity of the image, we reduce the number of samples and the amount of data generated by the time stretch camera in our proof-of-concept experiments by about three times. Optical data compression addresses the big data predicament in such systems.

Optically amplified detection for biomedical sensing and imaging.

Optical sensing and imaging methods for biomedical applications, such as spectroscopy and laser-scanning fluorescence microscopy, are incapable of performing sensitive detection at high scan rates due to the fundamental trade-off between sensitivity and speed. This is because fewer photons are detected during short integration times and hence the signal falls below the detector noise. Optical postamplification can, however, overcome this challenge by amplifying the collected optical signal after collection and before photodetection. Here we present a theoretical analysis of the sensitivity of high-speed biomedical sensing and imaging systems enhanced by optical postamplifiers. As a case study, we focus on Raman amplifiers because they produce gain at any wavelength within the gain medium's transparency window and are hence suitable for biomedical applications. Our analytical model shows that when limited by detector noise, such optically postamplified systems can achieve a sensitivity improvement of up to 20 dB in the visible to near-infrared spectral range without sacrificing speed. This analysis is expected to be valuable for design of fast real-time biomedical sensing and imaging systems.

Label-free high-throughput cell screening in flow.

Flow cytometry is a powerful tool for cell counting and biomarker detection in biotechnology and medicine especially with regards to blood analysis. Standard flow cytometers perform cell type classification both by estimating size and granularity of cells using forward- and side-scattered light signals and through the collection of emission spectra of fluorescently-labeled cells. However, cell surface labeling as a means of marking cells is often undesirable as many reagents negatively impact cellular viability or provide activating/inhibitory signals, which can alter the behavior of the desired cellular subtypes for downstream applications or analysis. To eliminate the need for labeling, we introduce a label-free imaging-based flow cytometer that measures size and cell protein concentration simultaneously either as a stand-alone instrument or as an add-on to conventional flow cytometers. Cell protein concentration adds a parameter to cell classification, which improves the specificity and sensitivity of flow cytometers without the requirement of cell labeling. This system uses coherent dispersive Fourier transform to perform phase imaging at flow speeds as high as a few meters per second.

Spectrally encoded angular light scattering.

The angular light scattering profile of microscopic particles significantly depends on their morphological parameters, such as size and shape. This dependency is widely used in state-of-the-art flow cytometry methods for particle classification. We introduce a new spectrally encoded angular light scattering method, with potential application in scanning flow cytometry. We show that a one-to-one wavelength-to-angle mapping enables the measurement of the angular dependence of scattered light from microscopic particles over a wide dynamic range. Improvement in dynamic range is obtained by equalizing the angular dependence of scattering via wavelength equalization. Continuous angular spectrum is obtained without mechanical scanning enabling single-shot measurement. Using this information, particle morphology can be determined with improved accuracy. We derive and experimentally verify an analytic wavelength-to-angle mapping model, facilitating rapid data processing. As a proof of concept, we demonstrate the method's capability of distinguishing differently sized polystyrene beads. The combination of this technique with time-stretch dispersive Fourier transform offers real-time and high-throughput (high frame rate) measurements and renders the method suitable for integration in standard flow cytometers.

Nomarski serial time-encoded amplified microscopy for high-speed contrast-enhanced imaging of transparent media.

High-speed high-contrast imaging modalities that enable image acquisition of transparent media without the need for chemical staining are essential tools for a broad range of applications; from semiconductor process monitoring to blood screening. Here we introduce a method for contrast-enhanced imaging of unstained transparent objects that is capable of high-throughput imaging. This method combines the Nomarski phase contrast capability with the ultrahigh frame rate and shutter speed of serial time-encoded amplified microscopy. As a proof of concept, we show imaging of a transparent test structure and white blood cells in flow at a shutter speed of 33 ps and a frame rate of 36.1 MHz using a single-pixel photo-detector. This method is expected to be a valuable tool for high-throughput screening of unstained cells.