High-performance reconstruction method combining total variation with a video denoiser for compressed ultrafast imaging.

Compressed ultrafast photography (CUP) is a novel two-dimensional (2D) imaging technique to capture ultrafast dynamic scenes. Effective image reconstruction is essential in CUP systems. However, existing reconstruction algorithms mostly rely on image priors and complex parameter spaces. Therefore, in general, they are time-consuming and result in poor imaging quality, which limits their practical applications. In this paper, we propose a novel reconstruction algorithm, to the best of our knowledge, named plug-in-plug-fast deep video denoising net-total variation (PnP-TV-FastDVDnet), which exploits an image's spatial features and correlation features in the temporal dimension. Therefore, it offers higher-quality images than those in previously reported methods. First, we built a forward mathematical model of the CUP, and the closed-form solution of the three suboptimization problems was derived according to plug-in and plug-out frames. Secondly, we used an advanced video denoising algorithm based on a neural network named FastDVDnet to solve the denoising problem. The peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) are improved on actual CUP data compared with traditional algorithms. On benchmark and real CUP datasets, the proposed method shows the comparable visual results while reducing the running time by 96% over state-of-the-art algorithms.

Handheld snapshot multi-spectral camera at tens-of-megapixel resolution.

Multi-spectral imaging is a fundamental tool characterizing the constituent energy of scene radiation. However, current multi-spectral video cameras cannot scale up beyond megapixel resolution due to optical constraints and the complexity of the reconstruction algorithms. To circumvent the above issues, we propose a tens-of-megapixel handheld multi-spectral videography approach (THETA), with a proof-of-concept camera achieving 65-megapixel videography of 12 wavebands within visible light range. The high performance is brought by multiple designs: We propose an imaging scheme to fabricate a thin mask for encoding spatio-spectral data using a conventional film camera. Afterwards, a fiber optic plate is introduced for building a compact prototype supporting pixel-wise encoding with a large space-bandwidth product. Finally, a deep-network-based algorithm is adopted for large-scale multi-spectral data decoding, with the coding pattern specially designed to facilitate efficient coarse-to-fine model training. Experimentally, we demonstrate THETA's advantageous and wide applications in outdoor imaging of large macroscopic scenes.

A Novel Reconstruction Algorithm with High Performance for Compressed Ultrafast Imaging.

Compressed ultrafast photography (CUP) is a type of two-dimensional (2D) imaging technique to observe ultrafast processes. Intelligence reconstruction methods that influence the imaging quality are an essential part of a CUP system. However, existing reconstruction algorithms mostly rely on image priors and complex parameter spaces. Therefore, it usually takes a lot of time to obtain acceptable reconstruction results, which limits the practical application of the CUP. In this paper, we proposed a novel reconstruction algorithm named PnP-FFDNet, which can provide a high quality and high efficiency compared to previous methods. First, we built a forward model of the CUP and three sub-optimization problems were obtained using the alternating direction multiplier method (ADMM), and the closed-form solution of the first sub-optimization problem was derived. Secondly, inspired by the PnP-ADMM framework, we used an advanced denoising algorithm based on a neural network named FFDNet to solve the second sub-optimization problem. On the real CUP data, PSNR and SSIM are improved by an average of 3 dB and 0.06, respectively, compared with traditional algorithms. Both on the benchmark dataset and on the real CUP data, the proposed method reduces the running time by an average of about 96% over state-of-the-art algorithms, and show comparable visual results, but in a much shorter running time.

Dynamic non-line-of-sight imaging system based on the optimization of point spread functions.

Non-line-of-sight (NLOS) imaging reveals hidden objects reflected from diffusing surfaces or behind scattering media. NLOS reconstruction is usually achieved by computational deconvolution of time-resolved transient data from a scanning single-photon avalanche diode (SPAD) detection system. However, using such a system requires a lengthy acquisition, impossible for capturing dynamic NLOS scenes. We propose to use a novel SPAD array and an optimization-based computational method to achieve NLOS reconstruction of 20 frames per second (fps). The imaging system's high efficiency drastically reduces the acquisition time for each frame. The forward projection optimization method robustly reconstructs NLOS scenes from low SNR data collected by the SPAD array. Experiments were conducted over a wide range of dynamic scenes in comparison with confocal and phase-field methods. Under the same exposure time, the proposed algorithm shows superior performances among state-of-the-art methods. To better analyze and validate our system, we also used simulated scenes to validate the advantages through quantitative benchmarks such as PSNR, SSIM and total variation analysis. Our system is anticipated to have the potential to achieve video-rate NLOS imaging.

Single-shot compressed ultrafast photography based on U-net network.

The compressive ultrafast photography (CUP) has achieved real-time femtosecond imaging based on the compressive-sensing methods. However, the reconstruction performance usually suffers from artifacts brought by strong noise, aberration, and distortion, which prevents its applications. We propose a deep compressive ultrafast photography (DeepCUP) method. Various numerical simulations have been demonstrated on both the MNIST and UCF-101 datasets and compared with other state-of-the-art algorithms. The result shows that our DeepCUP has a superior performance in both PSNR and SSIM compared to previous compressed-sensing methods. We also illustrate the outstanding performance of the proposed method under system errors and noise in comparison to other methods.