Single-Shot Intensity- and Phase-Sensitive Compressive Sensing-Based Coherent Modulation Ultrafast Imaging.

Ultrafast imaging can capture the dynamic scenes with a nanosecond and even femtosecond temporal resolution. Complementarily, phase imaging can provide the morphology, refractive index, or thickness information that intensity imaging cannot represent. Therefore, it is important to realize the simultaneous ultrafast intensity and phase imaging for achieving as much information as possible in the detection of ultrafast dynamic scenes. Here, we report a single-shot intensity- and phase-sensitive compressive sensing-based coherent modulation ultrafast imaging technique, shortened as CS-CMUI, which integrates coherent modulation imaging, compressive imaging, and streak imaging. We theoretically demonstrate through numerical simulations that CS-CMUI can obtain both the intensity and phase information of the dynamic scenes with ultrahigh fidelity. Furthermore, we experimentally build a CS-CMUI system and successfully measure the intensity and phase evolution of a multimode Q-switched laser pulse and the dynamical behavior of laser ablation on an indium tin oxide thin film. It is anticipated that CS-CMUI enables a profound comprehension of ultrafast phenomena and promotes the advancement of various practical applications, which will have substantial impact on fundamental and applied sciences.

Spatial-temporal characterization of photoemission in a streak-mode dynamic transmission electron microscope.

A long-standing motivation driving high-speed electron microscopy development is to capture phase transformations and material dynamics in real time with high spatial and temporal resolution. Current dynamic transmission electron microscopes (DTEMs) are limited to nanosecond temporal resolution and the ability to capture only a few frames of a transient event. With the motivation to overcome these limitations, we present our progress in developing a streak-mode DTEM (SM-DTEM) and demonstrate the recovery of picosecond images with high frame sequence depth. We first demonstrate that a zero-dimensional (0D) SM-DTEM can provide temporal information on any local region of interest with a 0.37 µm diameter, a 20-GHz sampling rate, and 1200 data points in the recorded trace. We use this method to characterize the temporal profile of the photoemitted electron pulse, finding that it deviates from the incident ultraviolet laser pulse and contains an unexpected peak near its onset. Then, we demonstrate a two-dimensional (2D) SM-DTEM, which uses compressed-sensing-based tomographic imaging to recover a full spatiotemporal photoemission profile over a 1.85-µm-diameter field of view with nanoscale spatial resolution, 370-ps inter-frame interval, and 140-frame sequence depth in a 50-ns time window. Finally, a perspective is given on the instrumental modifications necessary to further develop this promising technique with the goal of decreasing the time to capture a 2D SM-DTEM dataset.

Swept coded aperture real-time femtophotography.

Single-shot real-time femtophotography is indispensable for imaging ultrafast dynamics during their times of occurrence. Despite their advantages over conventional multi-shot approaches, existing techniques confront restricted imaging speed or degraded data quality by the deployed optoelectronic devices and face challenges in the application scope and acquisition accuracy. They are also hindered by the limitations in the acquirable information imposed by the sensing models. Here, we overcome these challenges by developing swept coded aperture real-time femtophotography (SCARF). This computational imaging modality enables all-optical ultrafast sweeping of a static coded aperture during the recording of an ultrafast event, bringing full-sequence encoding of up to 156.3 THz to every pixel on a CCD camera. We demonstrate SCARF's single-shot ultrafast imaging ability at tunable frame rates and spatial scales in both reflection and transmission modes. Using SCARF, we image ultrafast absorption in a semiconductor and ultrafast demagnetization of a metal alloy.

Short-wave Infrared Photoluminescence Lifetime Mapping of Rare-Earth Doped Nanoparticles Using All-Optical Streak Imaging.

The short-wave infrared (SWIR) photoluminescence lifetimes of rare-earth doped nanoparticles (RENPs) have found diverse applications in fundamental and applied research. Despite dazzling progress in the novel design and synthesis of RENPs with attractive optical properties, existing optical systems for SWIR photoluminescence lifetime imaging are still considerably restricted by inefficient photon detection, limited imaging speed, and low sensitivity. To overcome these challenges, SWIR photoluminescence lifetime imaging microscopy using an all-optical streak camera (PLIMASC) is developed. Synergizing scanning optics and a high-sensitivity InGaAs CMOS camera, SWIR-PLIMASC has a 1D imaging speed of up to 138.9 kHz in the spectral range of 900-1700 nm, which quantifies the photoluminescence lifetime of RENPs in a single shot. A 2D photoluminescence lifetime map can be acquired by 1D scanning of the sample. To showcase the power of SWIR-PLIMASC, a series of core-shell RENPs with distinct SWIR photoluminescence lifetimes is synthesized. In particular, using Er3+ -doped RENPs, SWIR-PLIMASC enables multiplexed anti-counterfeiting. Leveraging Ho3+ -doped RENPs as temperature indicators, this system is applied to SWIR photoluminescence lifetime-based thermometry. Opening up a new avenue for efficient SWIR photoluminescence lifetime mapping, this work is envisaged to contribute to advanced materials characterization, information science, and biomedicine.

Tutorial on compressed ultrafast photography.

Significance: Compressed ultrafast photography (CUP) is currently the world's fastest single-shot imaging technique. Through the integration of compressed sensing and streak imaging, CUP can capture a transient event in a single camera exposure with imaging speeds from thousands to trillions of frames per second, at micrometer-level spatial resolutions, and in broad sensing spectral ranges. Aim: This tutorial aims to provide a comprehensive review of CUP in its fundamental methods, system implementations, biomedical applications, and prospect. Approach: A step-by-step guideline to CUP's forward model and representative image reconstruction algorithms is presented with sample codes and illustrations in Matlab and Python. Then, CUP's hardware implementation is described with a focus on the representative techniques, advantages, and limitations of the three key components-the spatial encoder, the temporal shearing unit, and the two-dimensional sensor. Furthermore, four representative biomedical applications enabled by CUP are discussed, followed by the prospect of CUP's technical advancement. Conclusions: CUP has emerged as a state-of-the-art ultrafast imaging technology. Its advanced imaging ability and versatility contribute to unprecedented observations and new applications in biomedicine. CUP holds great promise in improving technical specifications and facilitating the investigation of biomedical processes.

Temporal resolution of ultrafast compressive imaging using a single-chirped optical probe.

Ultrafast compressive imaging captures three-dimensional spatiotemporal information of transient events in a single shot. When a single-chirped optical probe is applied, the temporal information is obtained from the probe modulated in amplitude or phase using a direct frequency-time mapping method. Here, we extend the analysis of the temporal resolution of conventional one-dimensional ultrafast measurement techniques such as spectral interferometry to that in three-dimensional ultrafast compressive imaging. In this way, both the amplitude and phase of the probe are necessary for a full Fourier transform method, which obtains temporal information with an improved resolution determined by probe spectral bandwidth. The improved temporal resolution potentially enables ultrafast compressive imaging with an effective imaging speed at the quadrillion-frames-per-second level.

Optical single-pixel volumetric imaging by three-dimensional light-field illumination.

Three-dimensional single-pixel imaging (3D SPI) has become an attractive imaging modality for both biomedical research and optical sensing. 3D-SPI techniques generally depend on time-of-flight or stereovision principle to extract depth information from backscattered light. However, existing implementations for these two optical schemes are limited to surface mapping of 3D objects at depth resolutions, at best, at the millimeter level. Here, we report 3D light-field illumination single-pixel microscopy (3D-LFI-SPM) that enables volumetric imaging of microscopic objects with a near-diffraction-limit 3D optical resolution. Aimed at 3D space reconstruction, 3D-LFI-SPM optically samples the 3D Fourier spectrum by combining 3D structured light-field illumination with single-element intensity detection. We build a 3D-LFI-SPM prototype that provides an imaging volume of ∼390 × 390 × 3,800 µm3 and achieves 2.7-µm lateral resolution and better than 37-µm axial resolution. Its capability of 3D visualization of label-free optical absorption contrast is demonstrated by imaging single algal cells in vivo. Our approach opens broad perspectives for 3D SPI with potential applications in various fields, such as biomedical functional imaging.

Compressed ultrahigh-speed single-pixel imaging by swept aggregate patterns.

Single-pixel imaging (SPI) has emerged as a powerful technique that uses coded wide-field illumination with sampling by a single-point detector. Most SPI systems are limited by the refresh rates of digital micromirror devices (DMDs) and time-consuming iterations in compressed-sensing (CS)-based reconstruction. Recent efforts in overcoming the speed limit in SPI, such as the use of fast-moving mechanical masks, suffer from low reconfigurability and/or reduced accuracy. To address these challenges, we develop SPI accelerated via swept aggregate patterns (SPI-ASAP) that combines a DMD with laser scanning hardware to achieve pattern projection rates of up to 14.1 MHz and tunable frame sizes of up to 101×103 pixels. Meanwhile, leveraging the structural properties of S-cyclic matrices, a lightweight CS reconstruction algorithm, fully compatible with parallel computing, is developed for real-time video streaming at 100 frames per second (fps). SPI-ASAP allows reconfigurable imaging in both transmission and reflection modes, dynamic imaging under strong ambient light, and offline ultrahigh-speed imaging at speeds of up to 12,000 fps.

Multi-scale band-limited illumination profilometry for robust three-dimensional surface imaging at video rate.

Dynamic three-dimensional (3D) surface imaging by phase-shifting fringe projection profilometry has been widely implemented in diverse applications. However, existing techniques fall short in simultaneously providing the robustness in solving spatially isolated 3D objects, the tolerance of large variation in surface reflectance, and the flexibility of tunable working distances with meter-square-level fields of view (FOVs) at video rate. In this work, we overcome these limitations by developing multi-scale band-limited illumination profilometry (MS-BLIP). Supported by the synergy of dual-level intensity projection, multi-frequency fringe projection, and an iterative method for distortion compensation, MS-BLIP can accurately discern spatially separated 3D objects with highly varying reflectance. MS-BLIP is demonstrated by dynamic 3D imaging of a translating engineered box and a rotating vase. With an FOV of up to 1.7 m × 1.1 m and a working distance of up to 2.8 m, MS-BLIP is applied to capturing full human-body movements at video rate.

Single-shot compressed optical field topography.

Femtosecond lasers are powerful in studying matter's ultrafast dynamics within femtosecond to attosecond time scales. Drawing a three-dimensional (3D) topological map of the optical field of a femtosecond laser pulse including its spatiotemporal amplitude and phase distributions, allows one to predict and understand the underlying physics of light interaction with matter, whose spatially resolved transient dielectric function experiences ultrafast evolution. However, such a task is technically challenging for two reasons: first, one has to capture in single-shot and squeeze the 3D information of an optical field profile into a two-dimensional (2D) detector; second, typical detectors are only sensitive to intensity or amplitude information rather than phase. Here we have demonstrated compressed optical field topography (COFT) drawing a 3D map for an ultrafast optical field in single-shot, by combining the coded aperture snapshot spectral imaging (CASSI) technique with a global 3D phase retrieval procedure. COFT can, in single-shot, fully characterize the spatiotemporal coupling of a femtosecond laser pulse, and live stream the light-speed propagation of an air plasma ionization front, unveiling its potential applications in ultrafast sciences.

High-speed super-resolution imaging with compressive imaging-based structured illumination microscopy.

Structured illumination microscopy (SIM) has been widely applied to investigating fine structures of biological samples by breaking the optical diffraction limitation. So far, video-rate imaging has been obtained in SIM, but the imaging speed was still limited due to the reconstruction of a super-solution image through multi-sampling, which hindered the applications in high-speed biomedical imaging. To overcome this limitation, here we develop compressive imaging-based structured illumination microscopy (CISIM) by synergizing SIM and compressive sensing (CS). Compared with conventional SIM, CISIM can greatly improve the super-resolution imaging speed by extracting multiple super-resolution images from one compressed image. Based on CISIM, we successfully reconstruct the super-resolution images in biological dynamics, and analyze the effect factors of image reconstruction quality, which verify the feasibility of CISIM. CISIM paves a way for high-speed super-resolution imaging, which may bring technological breakthroughs and significant applications in biomedical imaging.

Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging.

Photoluminescence lifetime imaging of upconverting nanoparticles is increasingly featured in recent progress in optical thermometry. Despite remarkable advances in photoluminescent temperature indicators, existing optical instruments lack the ability of wide-field photoluminescence lifetime imaging in real time, thus falling short in dynamic temperature mapping. Here, we report video-rate upconversion temperature sensing in wide field using single-shot photoluminescence lifetime imaging thermometry (SPLIT). Developed from a compressed-sensing ultrahigh-speed imaging paradigm, SPLIT first records wide-field luminescence intensity decay compressively in two views in a single exposure. Then, an algorithm, built upon the plug-and-play alternating direction method of multipliers, is used to reconstruct the video, from which the extracted lifetime distribution is converted to a temperature map. Using the core/shell NaGdF4:Er3+,Yb3+/NaGdF4 upconverting nanoparticles as the lifetime-based temperature indicators, we apply SPLIT in longitudinal wide-field temperature monitoring beneath a thin scattering medium. SPLIT also enables video-rate temperature mapping of a moving biological sample at single-cell resolution.

Compressed ultrafast tomographic imaging by passive spatiotemporal projections.

Existing streak-camera-based two-dimensional (2D) ultrafast imaging techniques are limited by long acquisition time, the trade-off between spatial and temporal resolutions, and a reduced field of view. They also require additional components, customization, or active illumination. Here we develop compressed ultrafast tomographic imaging (CUTI), which passively records 2D transient events with a standard streak camera. By grafting the concept of computed tomography to the spatiotemporal domain, the operations of temporal shearing and spatiotemporal integration in a streak camera's data acquisition can be equivalently expressed as the spatiotemporal projection of an (x,y,t) datacube from a certain angle. Aided by a new, to the best of our knowledge, compressed-sensing reconstruction algorithm, the 2D transient event can be accurately recovered in a few measurements. CUTI is exhibited as a new imaging mode universally adaptable to most streak cameras. Implemented in an image-converter streak camera, CUTI captures the sequential arrival of two spatially modulated ultrashort ultraviolet laser pulses at 0.5 trillion frames per second. Applied to a rotating-mirror streak camera, CUTI records an amination of fast-bouncing balls at 5,000 frames per second.

Camera-free three-dimensional dual photography.

We report camera-free three-dimensional (3D) dual photography. Inspired by the linkage between fringe projection profilometry (FPP) and dual photography, we propose to implement coordinate mapping to simultaneously sense the direct component of the light transport matrix and the surface profiles of 3D objects. By exploiting Helmholtz reciprocity, dual photography and scene relighting can thus be performed on 3D images. To verify the proposed imaging method, we have developed a single-pixel imaging system based on two digital micromirror devices (DMDs). Binary cyclic S-matrix patterns and binary sinusoidal fringe patterns are loaded on each DMD for scene encoding and virtual fringe projection, respectively. Using this system, we have demonstrated viewing and relighting 3D images at user-selectable perspectives. Our work extends the conceptual scope and the imaging capability of dual photography.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging.

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

Single-shot stereo-polarimetric compressed ultrafast photography for light-speed observation of high-dimensional optical transients with picosecond resolution.

Simultaneous and efficient ultrafast recording of multiple photon tags contributes to high-dimensional optical imaging and characterization in numerous fields. Existing high-dimensional optical imaging techniques that record space and polarization cannot detect the photon's time of arrival owing to the limited speeds of the state-of-the-art electronic sensors. Here, we overcome this long-standing limitation by implementing stereo-polarimetric compressed ultrafast photography (SP-CUP) to record light-speed high-dimensional events in a single exposure. Synergizing compressed sensing and streak imaging with stereoscopy and polarimetry, SP-CUP enables video-recording of five photon tags (x, y, z: space; t: time of arrival; and ψ: angle of linear polarization) at 100 billion frames per second with a picosecond temporal resolution. We applied SP-CUP to the spatiotemporal characterization of linear polarization dynamics in early-stage plasma emission from laser-induced breakdown. This system also allowed three-dimensional ultrafast imaging of the linear polarization properties of a single ultrashort laser pulse propagating in a scattering medium.

A cellular endolysosome-modulating pore-forming protein from a toad is negatively regulated by its paralog under oxidizing conditions.

Endolysosomes are key players in cell physiology, including molecular exchange, immunity, and environmental adaptation. They are the molecular targets of some pore-forming aerolysin-like proteins (ALPs) that are widely distributed in animals and plants and are functionally related to bacterial toxin aerolysins. ßγ-CAT is a complex of an ALP (BmALP1) and a trefoil factor (BmTFF3) in the firebelly toad (Bombina maxima). It is the first example of a secreted endogenous pore-forming protein that modulates the biochemical properties of endolysosomes by inducing pore formation in these intracellular vesicles. Here, using a large array of biochemical and cell biology methods, we report the identification of BmALP3, a paralog of BmALP1 that lacks membrane pore-forming capacity. We noted that both BmALP3 and BmALP1 contain a conserved cysteine in their C-terminal regions. BmALP3 was readily oxidized to a disulfide bond-linked homodimer, and this homodimer then oxidized BmALP1 via disulfide bond exchange, resulting in the dissociation of ßγ-CAT subunits and the elimination of biological activity. Consistent with its behavior in vitro, BmALP3 sensed environmental oxygen tension in vivo, leading to modulation of ßγ-CAT activity. Interestingly, we found that this C-terminal cysteine site is well conserved in numerous vertebrate ALPs. These findings uncover the existence of a regulatory ALP (BmALP3) that modulates the activity of an active ALP (BmALP1) in a redox-dependent manner, a property that differs from those of bacterial toxin aerolysins.

Single-shot ultrafast imaging attaining 70 trillion frames per second.

Real-time imaging of countless femtosecond dynamics requires extreme speeds orders of magnitude beyond the limits of electronic sensors. Existing femtosecond imaging modalities either require event repetition or provide single-shot acquisition with no more than 1013 frames per second (fps) and 3 × 102 frames. Here, we report compressed ultrafast spectral photography (CUSP), which attains several new records in single-shot multi-dimensional imaging speeds. In active mode, CUSP achieves both 7 × 1013 fps and 103 frames simultaneously by synergizing spectral encoding, pulse splitting, temporal shearing, and compressed sensing-enabling unprecedented quantitative imaging of rapid nonlinear light-matter interaction. In passive mode, CUSP provides four-dimensional (4D) spectral imaging at 0.5 × 1012 fps, allowing the first single-shot spectrally resolved fluorescence lifetime imaging microscopy (SR-FLIM). As a real-time multi-dimensional imaging technology with the highest speeds and most frames, CUSP is envisioned to play instrumental roles in numerous pivotal scientific studies without the need for event repetition.

Real-time high-speed three-dimensional surface imaging using band-limited illumination profilometry with a CoaXPress interface.

High-speed three-dimensional (3D) surface imaging by structured-light profilometry is currently driven by numerous applications. However, the limited speeds in fringe pattern projection, image acquisition, and data transmission have strained the existing methods from reaching kilohertz-level acquisition, processing, and display of 3D information during the occurrence of dynamic events (i.e., in real time). To overcome these limitations, we have developed band-limited illumination profilometry (BLIP) with a CoaXPress interface (CI), which enables real-time high-speed 3D surface imaging. We have demonstrated the system's performance by imaging various static and fast-moving 3D objects in real time. We have also applied this system in fluid mechanics by imaging dynamics of a flag, which allowed observation of the wave propagation, gravity-induced phase mismatch, and asymmetric flapping motion. We expect CI-BLIP to find diverse scientific and industrial applications.

Picosecond-resolution phase-sensitive imaging of transparent objects in a single shot.

With the growing interest in the optical imaging of ultrafast phenomena in transparent objects, from shock wave to neuronal action potentials, high contrast imaging at high frame rates has become desirable. While phase sensitivity provides the contrast, the frame rates and sequence depths are highly limited by the detectors. Here, we present phase-sensitive compressed ultrafast photography (pCUP) for single-shot real-time ultrafast imaging of transparent objects by combining the contrast of dark-field imaging with the speed and the sequence depth of CUP. By imaging the optical Kerr effect and shock wave propagation, we demonstrate that pCUP can image light-speed phase signals in a single shot with up to 350 frames captured at up to 1 trillion frames per second. We expect pCUP to be broadly used for a vast range of fundamental and applied sciences.