Compressive video via IR-pulsed illumination.

We propose and demonstrate a compressive temporal imaging system based on pulsed illumination to encode temporal dynamics into the signal received by the imaging sensor during exposure time. Our approach enables >10x increase in effective frame rate without increasing camera complexity. To mitigate the complexity of the inverse problem during reconstruction, we introduce two keyframes: one before and one after the coded frame. We also craft what we believe to be a novel deep learning architecture for improved reconstruction of the high-speed scenes, combining specialized convolutional and transformer architectures. Simulation and experimental results clearly demonstrate the reconstruction of high-quality, high-speed videos from the compressed data.

Snapshot ptychography on array cameras.

We use convolutional neural networks to recover images optically down-sampled by 6.7 × using coherent aperture synthesis over a 16 camera array. Where conventional ptychography relies on scanning and oversampling, here we apply decompressive neural estimation to recover full resolution image from a single snapshot, although as shown in simulation multiple snapshots can be used to improve signal-to-noise ratio (SNR). In place training on experimental measurements eliminates the need to directly calibrate the measurement system. We also present simulations of diverse array camera sampling strategies to explore how snapshot compressive systems might be optimized.

Photon-limited bounds for phase retrieval.

We show that the optimal Cramér-Rao lower bound on the mean-square error for the estimation of a coherent signal from photon-limited intensity measurements is equal to the number of signal elements, or the number of signal elements minus one when we account for the unobservable reference phase. Whereas this bound is attained by phase-quadrature holography, we also show that it can be attained through a phase-retrieval system that does not require a coherent reference. We also present the bounds for classic phase-retrieval and ptychography, and show that practical coding strategies can approach optimal performance.

A modular hierarchical array camera.

Array cameras removed the optical limitations of a single camera and paved the way for high-performance imaging via the combination of micro-cameras and computation to fuse multiple aperture images. However, existing solutions use dense arrays of cameras that require laborious calibration and lack flexibility and practicality. Inspired by the cognition function principle of the human brain, we develop an unstructured array camera system that adopts a hierarchical modular design with multiscale hybrid cameras composing different modules. Intelligent computations are designed to collaboratively operate along both intra- and intermodule pathways. This system can adaptively allocate imagery resources to dramatically reduce the hardware cost and possesses unprecedented flexibility, robustness, and versatility. Large scenes of real-world data were acquired to perform human-centric studies for the assessment of human behaviours at the individual level and crowd behaviours at the population level requiring high-resolution long-term monitoring of dynamic wide-area scenes.

Review of bio-optical imaging systems with a high space-bandwidth product.

Optical imaging has served as a primary method to collect information about biosystems across scales-from functionalities of tissues to morphological structures of cells and even at biomolecular levels. However, to adequately characterize a complex biosystem, an imaging system with a number of resolvable points, referred to as a space-bandwidth product (SBP), in excess of one billion is typically needed. Since a gigapixel-scale far exceeds the capacity of current optical imagers, compromises must be made to obtain either a low spatial resolution or a narrow field-of-view (FOV). The problem originates from constituent refractive optics-the larger the aperture, the more challenging the correction of lens aberrations. Therefore, it is impractical for a conventional optical imaging system to achieve an SBP over hundreds of millions. To address this unmet need, a variety of high-SBP imagers have emerged over the past decade, enabling an unprecedented resolution and FOV beyond the limit of conventional optics. We provide a comprehensive survey of high-SBP imaging techniques, exploring their underlying principles and applications in bioimaging.

Noise suppression for ballistic-photons based on compressive in-line holographic imaging through an inhomogeneous medium.

Noise suppression is one of the most important tasks in imaging through inhomogeneous mediums. Here, we proposed a denoising approach based on compressive in-line holography for imaging through an inhomogeneous medium. A reference-beam-free system with a low-cost continuous-wave laser is presented. The suppression against the noise, which is brought by the scattering photons, is presented in simulations using the proposed algorithm. The noise immunity is demonstrated in lensless imaging behind a random phase mask with an optical depth of 1.42 by single exposure, as well as behind a ground glass with an optical depth of 6.38 by multiple exposures.

Rank Minimization for Snapshot Compressive Imaging.

Snapshot compressive imaging (SCI) refers to compressive imaging systems where multiple frames are mapped into a single measurement, with video compressive imaging and hyperspectral compressive imaging as two representative applications. Though exciting results of high-speed videos and hyperspectral images have been demonstrated, the poor reconstruction quality precludes SCI from wide applications. This paper aims to boost the reconstruction quality of SCI via exploiting the high-dimensional structure in the desired signal. We build a joint model to integrate the nonlocal self-similarity of video/hyperspectral frames and the rank minimization approach with the SCI sensing process. Following this, an alternating minimization algorithm is developed to solve this non-convex problem. We further investigate the special structure of the sampling process in SCI to tackle the computational workload and memory issues in SCI reconstruction. Both simulation and real data (captured by four different SCI cameras) results demonstrate that our proposed algorithm leads to significant improvements compared with current state-of-the-art algorithms. We hope our results will encourage the researchers and engineers to pursue further in compressive imaging for real applications.

Twin-Image-Free Holography: A Compressive Sensing Approach.

Holographic reconstruction is troubled by the phase-conjugate wave front arising from Hermitian symmetry of the complex field. The so-called twin image obfuscates the reconstruction in solving the inverse problem. Here we quantitatively reveal how and how much the twin image affects the reconstruction and propose a compressive sensing (CS) approach to reconstruct a hologram completely free from the twin image. Using the canonical basis, the incoherence condition of CS is naturally satisfied by the Fourier transformation associated with wave propagation. With the propagation kernel function related to the distance, the object wave diffracts into a sharp pattern while the phase-conjugate wave diffracts into a diffuse pattern. An iterative algorithm using a total variation sparsity constraint could filter out the diffuse conjugated signal and overcome the inherent physical symmetry of holographic reconstruction. The feasibility is verified by simulation and experimental results, as well as a comparative study to an existing phase retrieval method.

Field of view in monocentric multiscale cameras.

Conventionally, the field of view of a camera is understood as the angular extent of a convex circular or rectangular region. Parallel camera architectures with computational image stitching, however, allow implementation of a field of view with an arbitrary shape. Monocentric multiscale lenses further allow the implementation of an arbitrary field of view in camera volumes comparable to conventional single-lens systems. In contrast with conventional wide-field-of-view systems, multiscale design can also achieve nearly uniform resolution across the entire field of view. This paper presents several design studies obtaining unconventional fields of view using this approach.

Efficient block-wise algorithm for compressive holography.

Compressive holography is a relatively time-consuming image estimation in convex optimized problem. We propose an efficient block-wise algorithm to limit the searching space and reduce the calculation time while keeping the reconstruction quality. The effective anti-aliasing boundary of the sub-hologram is located to determine the block size for compressive reconstruction in the total-variation two-step iterative shrinkage/thresholding algorithm. Padded sub-holograms could be reconstructed in parallel by using multi-core processors. Compared with the traditional compressive holography, the block-wise algorithm could take approximately 1/50 of the reconstruction time and achieve an improved reconstruction quality.

Galilean monocentric multiscale optical systems.

The first generation of monocentric multiscale gigapixel cameras used Keplerian designs to enable full field coverage. This paper considers alternative designs that remove the requirement that adjacent subimages overlap. Removing this constraint enables Galilean designs that reduce system volume and improve relative illumination and image quality. The entrance aperture can also be moved to more closely approximate telecentricity and gaps in the field of view can be filled using multiple co-boresighted MMS cameras. Even with multiple cameras, Galilean systems can still reduce the total volume by 10 times relative to previous Keplerian designs.

Heterogeneous camera array for multispectral light field imaging.

Multispectral light field acquisition is challenging due to the increased dimensionality of the problem. In this paper, inspired by anaglyph theory (i.e. the ability of human eyes to synthesize colored stereo perception from color-complementary (such as red and cyan) views), we propose to capture the multispectral light field using multiple cameras with different wide band filters. A convolutional neural network is used to extract the joint information of different spectral channels and to pair the cross-channel images. In our experiment, results on both synthetic data and real data captured by our prototype system validate the effectiveness and accuracy of proposed method.

Coded Apertures in Mass Spectrometry.

The use of coded apertures in mass spectrometry can break the trade-off between throughput and resolution that has historically plagued conventional instruments. Despite their very early stage of development, coded apertures have been shown to increase throughput by more than one order of magnitude, with no loss in resolution in a simple 90-degree magnetic sector. This enhanced throughput can increase the signal level with respect to the underlying noise, thereby significantly improving sensitivity to low concentrations of analyte. Simultaneous resolution can be maintained, preventing any decrease in selectivity. Both one- and two-dimensional (2D) codes have been demonstrated. A 2D code can provide increased measurement diversity and therefore improved numerical conditioning of the mass spectrum that is reconstructed from the coded signal. This review discusses the state of development, the applications where coding is expected to provide added value, and the various instrument modifications necessary to implement coded apertures in mass spectrometers.

Compressive sensing and adaptive sampling applied to millimeter wave inverse synthetic aperture imaging.

In order to improve speed and efficiency over traditional scanning methods, a Bayesian compressive sensing algorithm using adaptive spatial sampling is developed for single detector millimeter wave synthetic aperture imaging. The application of this algorithm is compared to random sampling to demonstrate that the adaptive algorithm converges faster for simple targets and generates more reliable reconstructions for complex targets.

Snapshot fan beam coded aperture coherent scatter tomography.

We use coherently scattered X-rays to measure the molecular composition of an object throughout its volume. We image a planar slice of the object in a single snapshot by illuminating it with a fan beam and placing a coded aperture between the object and the detectors. We characterize the system and demonstrate a resolution of 13 mm in range and 2 mm in cross-range and a fractional momentum transfer resolution of 15%. In addition, we show that this technique allows a 100x speedup compared to previously-studied pencil beam systems using the same components. Finally, by scanning an object through the beam, we image the full 4-dimensional data cube (3 spatial and 1 material dimension) for complete volumetric molecular imaging.

Spectral-temporal compressive imaging.

This Letter presents a compressive camera that integrates mechanical translation and spectral dispersion to compress a multi-spectral, high-speed scene onto a monochrome, video-rate detector. Experimental reconstructions of 17 spectral channels and 11 temporal channels from a single measurement are reported for a megapixel-scale monochrome camera.

Single-sensor multispeaker listening with acoustic metamaterials.

Designing a "cocktail party listener" that functionally mimics the selective perception of a human auditory system has been pursued over the past decades. By exploiting acoustic metamaterials and compressive sensing, we present here a single-sensor listening device that separates simultaneous overlapping sounds from different sources. The device with a compact array of resonant metamaterials is demonstrated to distinguish three overlapping and independent sources with 96.67% correct audio recognition. Segregation of the audio signals is achieved using physical layer encoding without relying on source characteristics. This hardware approach to multichannel source separation can be applied to robust speech recognition and hearing aids and may be extended to other acoustic imaging and sensing applications.

Order of Magnitude Signal Gain in Magnetic Sector Mass Spectrometry Via Aperture Coding.

Miniaturizing instruments for spectroscopic applications requires the designer to confront a tradeoff between instrument resolution and instrument throughput [and associated signal-to-background-ratio (SBR)]. This work demonstrates a solution to this tradeoff in sector mass spectrometry by the first application of one-dimensional (1D) spatially coded apertures, similar to those previously demonstrated in optics. This was accomplished by replacing the input slit of a simple 90° magnetic sector mass spectrometer with a specifically designed coded aperture, deriving the corresponding forward mathematical model and spectral reconstruction algorithm, and then utilizing the resulting system to measure and reconstruct the mass spectra of argon, acetone, and ethanol. We expect the application of coded apertures to sector instrument designs will lead to miniature mass spectrometers that maintain the high performance of larger instruments, enabling field detection of trace chemicals and point-of-use mass spectrometry.

Spatial light modulator based color polarization imaging.

We describe a compressive snapshot color polarization imager that encodes spatial, spectral, and polarization information using a liquid crystal modulator. We experimentally show that polarization imaging is compressible by multiplexing polarization states and present the reconstruction results. This compressive camera captures the spatial distribution of four polarizations and three color channels. It achieves <0.027° spatial resolution, 10(3) average extinction ratio, and >30 PSNR.

Compressive sensing by learning a Gaussian mixture model from measurements.

Compressive sensing of signals drawn from a Gaussian mixture model (GMM) admits closed-form minimum mean squared error reconstruction from incomplete linear measurements. An accurate GMM signal model is usually not available a priori, because it is difficult to obtain training signals that match the statistics of the signals being sensed. We propose to solve that problem by learning the signal model in situ, based directly on the compressive measurements of the signals, without resorting to other signals to train a model. A key feature of our method is that the signals being sensed are treated as random variables and are integrated out in the likelihood. We derive a maximum marginal likelihood estimator (MMLE) that maximizes the likelihood of the GMM of the underlying signals given only their linear compressive measurements. We extend the MMLE to a GMM with dominantly low-rank covariance matrices, to gain computational speedup. We report extensive experimental results on image inpainting, compressive sensing of high-speed video, and compressive hyperspectral imaging (the latter two based on real compressive cameras). The results demonstrate that the proposed methods outperform state-of-the-art methods by significant margins.