Phase unwrapping for phase imaging using the plug-and-play proximal algorithm.

Phase unwrapping (PU) is essential for various scientific optical applications. This process aims to estimate continuous phase values from acquired wrapped values, which are limited to the interval (-π,π]. However, the PU process can be challenging due to factors such as insufficient sampling, measurement errors, and inadequate equipment calibration, which can introduce excessive noise and unexpected phase discontinuities. This paper presents a robust iterative method based on the plug-and-play (PnP) proximal algorithm to unwrap two-dimensional phase values while simultaneously removing noise at each iteration. Using a least-squares formulation based on local phase differences and reformulating it as a partially differentiable equation, it is possible to employ the fast cosine transform to obtain a closed-form solution for one of the subproblems within the PnP framework. As a result, reliable phase reconstruction can be achieved even in scenarios with extremely high noise levels.

Learning Time-multiplexed phase-coded apertures for snapshot spectral-depth imaging.

Depth and spectral imaging are essential technologies for a myriad of applications but have been conventionally studied as individual problems. Recent efforts have been made to optically encode spectral-depth (SD) information jointly in a single image sensor measurement, subsequently decoded by a computational algorithm. The performance of single snapshot SD imaging systems mainly depends on the optical modulation function, referred to as codification, and the computational methods used to recover the SD information from the coded measurement. The optical modulation has been conventionally realized using coded apertures (CAs), phase masks, prisms or gratings, active illumination, and many others. In this work, we propose an optical modulation (codification) strategy that employs a color-coded aperture (CCA) in conjunction with a time-varying phase-coded aperture and a spatially-varying pixel shutter, thus yielding an effective time-multiplexed coded aperture (TMCA). We show that the proposed TMCA entails a spatially-variant point spread function (PSF) for a constant depth in a scene, which, in turn, facilitates the distinguishability, and therefore, better recovery of the depth information. Further, the selective filtering of specific spectral bands by the CCA encodes relevant spectral information that is disentangled using a reconstruction algorithm. We leverage the advances of deep learning techniques to jointly learn the optical modulation and the computational decoding algorithm in an end-to-end (E2E) framework. We demonstrate via simulations and with a real testbed prototype that the proposed TMCA strategy outperforms state-of-the-art snapshot SD imaging alternatives in both spectral and depth reconstruction quality.

Computational spectral imaging: a contemporary overview.

Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow the identification of objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared with conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarizes the advances in CSI, starting with SI and its relevance and continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, as well as the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks.

Calibration reinforcement regularizations for optimized snapshot spectral imaging.

Optical coding is a fundamental tool in snapshot computational spectral imaging for capturing encoded scenes that are then decoded by solving an inverse problem. Optical encoding design is crucial, as it determines the invertibility properties of the system sensing matrix. To ensure a realistic design, the optical mathematical forward model must match the physical sensing. However, stochastic variations related to non-ideal characteristics of the implementation exist; therefore, these variables are not known a priori and have to be calibrated in the laboratory setup. Thus, the optical encoding design leads to suboptimal performance in practice, even if an exhaustive calibration process is carried out. This work proposes an algorithm to speed up the reconstruction process in a snapshot computational spectral imaging, in which theoretically optimized coding design is distorted by the implementation process. Specifically, two regularizers are proposed that perform the gradient algorithm iterations of the distorted calibrated system in the direction of the originally, theoretically optimized system. We illustrate the benefits of the reinforcement regularizers for several state-of-the-art recovery algorithms. For a given lower bound performance, the algorithm converges in fewer iterations due to the effect of the regularizers. Simulation results show an improvement of up to 2.5 dB of peak signal-to-noise ratio (PSNR) when fixing the number of iterations. Furthermore, the required number of iterations reduces up to 50% when the proposed regularizers are included to obtain a desired performance quality. Finally, the effectiveness of the proposed reinforcement regularizations was evaluated in a test-bed implementation, where a better spectral reconstruction was evidenced when compared with a non-regularized system's reconstruction.

Hyperspectral camera as a compact payload architecture for remote sensing applications.

Monitoring and observation over the surface of the Earth have been a matter of global interest. In this path, recent efforts aim to develop a spatial mission to perform remote sensing applications. Mainly, CubeSat nanosatellites have emerged as a standard for developing low-weight and small-sized instruments. In terms of payloads, state-of-the-art optical systems for CubeSats are expensive and designed to work in general use cases. To overcome these limitations, this paper presents a 1.4 U compact optical system to acquire spectral images from a CubeSat standard satellite at the height of 550 km. To validate the proposed architecture, optical simulations using ray tracing simulation software are presented. Because the performance of computer vision tasks is highly related to data quality, we compared the optical system in terms of the classification performance on a real remote sensing application. The performances of the optical characterization and land cover classification show that the proposed optical system achieves a compact instrument, operating at a spectral range from 450 nm to 900 nm discretized on 35 spectral bands. The optical system has an overall f-number of 3.41 with a ground sampling distance of 52.8 m and a swath of 40 km. Additionally, the design parameters for each optical element are publicly available for validation, repeatability, and reproducibility of the results.

C-3SPCD: coded aperture similarity constrained design for spatio-spectral classification of single-pixel measurements.

In recent years, compressive spectral imaging (CSI) has emerged as a new acquisition technique that acquires coded projections of the spectral scene, reducing considerably storage and transmission costs. Among several CSI devices, the single-pixel camera (SPC) architecture excels due to its low implementation cost when acquiring a large number of spectral bands. Although CSI allows efficient sampling, a complete reconstruction of the underlying scene is needed to perform any processing task, which involves solving a computationally expensive optimization problem. In this paper, we propose a fast method to classify the underlying spectral image by directly using compressed SPC measurements, avoiding reconstruction. In particular, the proposed method acquires an RGB image of the scene as side information to design the SPC coding patterns. Our design approach allows incorporating the similarity information of neighboring pixels from the RGB image into compressed measurements. After acquiring the compressed measurements with our designed coding patterns, we extract features of the scene to perform classification without reconstruction. After simulations, we obtained an overall accuracy of 95.41% and 97.72% for the Pavia University and Salinas spectral images, respectively. Furthermore, we tested our approach in the laboratory and classified our own dataset, which has four different materials: flowers, sand, grass, and dry leaves, with an overall accuracy of 94.66%.

Learning spectral initialization for phase retrieval via deep neural networks.

Phase retrieval (PR) arises from the lack of phase information in the measures recorded by optical sensors. Phase masks that modulate the optical field and reduce ambiguities in the PR problem by producing redundancy in coded diffraction patterns (CDPs) have been included in these diffractive optical systems. Several algorithms have been developed to solve the PR problem from CDPs. Also, deep neural networks (DNNs) are used for solving inverse problems in computational imaging by considering physical constraints in propagation models. However, traditional algorithms based on non-convex formulation include an initialization stage that requires a high number of iterations to properly estimate the optical field. This work proposes an end-to-end (E2E) approach for addressing the PR problem, which jointly learns the spectral initialization and network parameters. Mainly, the proposed deep network approach contains an optical layer that simulates the propagation model in diffractive optical systems, an initialization layer that approximates the underlying optical field from CDPs, and a double branch DNN that improves the obtained initial guess by separately recovering phase and amplitude information. Simulation results show that the proposed E2E approach for PR requires fewer snapshots and iterations than the state of the art.

JR2net: a joint non-linear representation and recovery network for compressive spectral imaging.

Deep learning models are state-of-the-art in compressive spectral imaging (CSI) recovery. These methods use a deep neural network (DNN) as an image generator to learn non-linear mapping from compressed measurements to the spectral image. For instance, the deep spectral prior approach uses a convolutional autoencoder (CAE) network in the optimization algorithm to recover the spectral image by using a non-linear representation. However, the CAE training is detached from the recovery problem, which does not guarantee optimal representation of the spectral images for the CSI problem. This work proposes a joint non-linear representation and recovery network (JR2net), linking the representation and recovery task into a single optimization problem. JR2net consists of an optimization-inspired network following an alternating direction method of multipliers (ADMM) formulation that learns a non-linear low-dimensional representation and simultaneously performs the spectral image recovery, trained via the end-to-end approach. Experimental results show the superiority of the proposed method with improvements up to 2.57 dB in peak signal-to-noise ratio (PSNR) and performance around 2000 times faster than state-of-the-art methods.

Snapshot compressive spectral depth imaging from coded aberrations.

Compressive spectral depth imaging (CSDI) is an emerging technology aiming to reconstruct spectral and depth information of a scene from a limited set of two-dimensional projections. CSDI architectures have conventionally relied on stereo setups that require the acquisition of multiple shots attained via dynamically programmable spatial light modulators (SLM). This work proposes a snapshot CSDI architecture that exploits both phase and amplitude modulation and uses a single image sensor. Specifically, we modulate the spectral-depth information in two steps. Firstly, a deformable mirror (DM) is used as a phase modulator to induce a focal length sweeping while simultaneously introducing a controlled aberration. The phase-modulated wavefront is then spatially modulated and spectrally dispersed by a digital micromirror device (DMD) and a prism, respectively. Therefore, each depth plane is modulated by a variable phase and binary code. Complimentary, we also propose a computational methodology to recover the underlying spectral depth hypercube efficiently. Through simulations and our experimental proof-of-concept implementation, we demonstrate that the proposed computational imaging system is a viable approach to capture spectral-depth hypercubes from a single image.

Compressive spectral image reconstruction using deep prior and low-rank tensor representation.

Compressive spectral imaging (CSI) has emerged as an alternative spectral image acquisition technology, which reduces the number of measurements at the cost of requiring a recovery process. In general, the reconstruction methods are based on handcrafted priors used as regularizers in optimization algorithms or recent deep neural networks employed as an image generator to learn a non-linear mapping from the low-dimensional compressed measurements to the image space. However, these deep learning methods need many spectral images to obtain good performance. In this work, a deep recovery framework for CSI without training data is presented. The proposed method is based on the fact that the structure of some deep neural networks and an appropriated low-dimensional structure are sufficient to impose a structure of the underlying spectral image from CSI. We analyzed the low-dimensional structure via the Tucker representation, modeled in the first net layer. The proposed scheme is obtained by minimizing the ${\ell _2}$-norm distance between the compressive measurements and the predicted measurements, and the desired recovered spectral image is formed just before the forward operator. Simulated and experimental results verify the effectiveness of the proposed method for the coded aperture snapshot spectral imaging.

High-resolution coded aperture optimization for super-resolved compressive x-ray cone-beam computed tomography.

Compressive x-ray cone-beam computed tomography (CBCT) approaches rely on coded apertures (CA) along multiple view angles to block a portion of the x-ray energy traveling towards the detectors. Previous work has shown that designing CA patterns yields improved images. Most designs, however, are focused on multi-shot fan-beam (FB) systems, handling a 1:1 ratio between CA features and detector elements. In consequence, image resolution is subject to the detector pixel size. Moreover, CA optimization for computed tomography involves strong binarization assumptions, impractical data rearrangements, or computationally expensive tasks such as singular value decomposition (SVD). Instead of using higher-resolution CA distributions in a multi-slice system with a more dense detector array, this work presents a method for designing the CA patterns in a compressive CBCT system under a super-resolution configuration, i.e., high-resolution CA patterns are designed to obtain high-resolution images from lower-resolution projections. The proposed method takes advantage of the Gershgorin theorem since its algebraic interpretation relates the circle radii with the eigenvalue bounds, whose minimization improves the condition of the system matrix. Simulations with medical data sets show that the proposed design attains high-resolution images from lower-resolution detectors in a single-shot CBCT scenario. Besides, image quality is improved in up to 5 dB of peak signal-to-noise compared to random CA patterns for different super-resolution factors. Moreover, reconstructions from Monte Carlo simulated projections show up to 3 dB improvements. Further, for the analyzed cases, the computational load of the proposed approach is up to three orders of magnitude lower than that of SVD-based methods.

Coupled deep learning coded aperture design for compressive image classification.

A coupled deep learning approach for coded aperture design and single-pixel measurements classification is proposed. A whole neural network is trained to simultaneously optimize the binary sensing matrix of a single-pixel camera (SPC) and the parameters of a classification network, considering the constraints imposed by the compressive architecture. Then, new single-pixel measurements can be acquired and classified with the learned parameters. This method avoids the reconstruction process while maintaining classification reliability. In particular, two network architectures were proposed, one learns re-projected measurements to the image size, and the other extracts small features directly from the compressive measurements. They were simulated using two image data sets and a test-bed implementation. The first network beats in around 10% the accuracy reached by the state-of-the-art methods. A 2x increase in computing time is achieved with the second proposed net.

SPUD: simultaneous phase unwrapping and denoising algorithm for phase imaging.

Recent methods for phase unwrapping in the presence of noise include denoising algorithms to filter out noise as a preprocessing stage. However, including a denoising stage increases the overall computational complexity resulting in long execution times. In this paper, we present a noniterative simultaneous phase unwrapping and denoising algorithm for phase imaging, referred to as SPUD. The proposed method relies on the least squares discrete cosine transform (DCT) solution for phase unwrapping with an additional sparsity constraint on the DCT coefficients of the unwrapped solution. Simulation results with different levels of noise and wrapped phase fringe density reveal the suitability of the proposed method for accurate phase unwrapping and restoration. When compared to the 2D windowed Fourier transform filter, SPUD performs better in terms of phase error and execution times. The processing of experimental data from synthetic aperture radar showed the capability for processing real images, including removing phase dislocations. An implementation of the proposed algorithm can be accessed and executed through a Code Ocean compute capsule.

Compressive spectral imaging via deformable mirror and colored-mosaic detector.

Compressive Spectral Imaging (CSI) is an emerging technology that aims at reconstructing a spectral image from a limited set of two-dimensional projections. To capture these projections, CSI architectures often combine light dispersive elements with coded apertures or programmable spatial light modulators. This work introduces a novel and compact CSI architecture based on a deformable mirror and a colored-filter detector to produce compressive spatio-spectral projections without the need of a grating or prism. Alongside, we propose a tensor-based reconstruction algorithm to recover the spatial-spectral information from the compressed measurements. Experimental results on both simulated and real datasets demonstrate efficacy of the proposed acquisition architecture and the especially crafted inversion algorithms.

Supervised spatio-spectral classification of fused images using superpixels.

The low spatial resolution of hyperspectral (HS) images generally limits the classification accuracy. Therefore, different multiresolution data fusion techniques have been proposed in the literature. In this paper, a method for supervised classification of spectral images from data fusion measurements is proposed. Specifically, the proposed approach exploits the spatial information of an RGB image by grouping pixels with similar characteristics into superpixels and fuses such features with the spectral information of an HS image. Simulations results on three datasets show that the proposed classification method improves the overall accuracy and reduces the computational complexity compared to the traditional approach that first performs the fusion followed by the classification.

Shifting colored coded aperture design for spectral imaging.

Compressive spectral imaging (CSI) systems sense 3D spatio-spectral data cubes with just a few two-dimensional (2D) projections by using a coded aperture, a dispersive element, and a focal plane array (FPA). The coded apertures in these systems, whose main function is the modulation of the data cube, are often implemented through photomasks attached to piezoelectric devices. A remarkable improvement on this configuration has been recently proposed, the replacement of the block-unblock coded apertures by patterned optical filter arrays, referred to as "colored" coded apertures, which allow spatial and spectral modulation. When using these colored coded apertures, its real implementation in terms of cost and complexity directly depends on the number of filters to be used, as well as the number of shots to be captured. A shifting colored coded aperture optimization featuring these observations is proposed, with the aim to improve the imaging quality reconstruction and to generate an achievable optical implementation with a limited number of filters requiring only one mask to acquire any number of shots. The mathematical model of the computational imaging strategy to overcome the practical limitations of actual CSI systems is presented along with a testbed implementation. Simulations, as well as experimental results, will prove the accuracy and performance of the proposed shifting colored coded aperture design over the current literature designs.

Adaptive filter design via a gradient thresholding algorithm for compressive spectral imaging.

Sensing a spectral image data cube has traditionally been a time-consuming task since it requires a scanning process. In contrast, compressive spectral imaging (CSI) has attracted widespread interest since it requires fewer samples than scanning systems to acquire the data cube, thus improving the sensing speed. CSI captures linear projections of the scene, and then a reconstruction algorithm estimates the underlying scene. One notable CSI architecture is the color coded aperture snapshot spectral imager (C-CASSI), which employs pixelated filter arrays as the coding patterns to spatially and spectrally encode the incoming light. Up to date works on C-CASSI have used non-adaptive color coded apertures. Non-adaptive sampling ignores prior information about the signal to design the coding patterns. Therefore, this work proposes a method to adaptively design the color coded aperture, such that the quality of image reconstruction is improved. In more detail, this work introduces a gradient thresholding algorithm, which computes the consecutive color coded aperture from a rapidly reconstructed low-resolution version of the data cube. The successive adaptive patterns enable recovering a data cube in the presence of Gaussian noise with higher image quality. Real reconstructions and simulations evidence an improvement of up to 3 dB in the quality of image reconstruction of the proposed method in comparison with state-of-the-art non-adaptive techniques.

Joint sparse and low rank recovery algorithm for compressive hyperspectral imaging.

Compressive spectral imaging techniques encode and disperse a hyperspectral image (HSI) to sense its spatial and spectral information with few bidimensional (2D) multiplexed projections. Recovering the original HSI from the 2D projections is carried by traditional compressive sensing-based techniques that exploit the sparsity property of natural HSI as they are represented in a proper orthonormal basis. Nevertheless, HSIs also exhibit a low rank property inasmuch only a few numbers of spectral signatures are present in the images. Specifically, when an HSI is rearranged as a matrix whose columns represent vectorized 2D spatial images in a different wavelength, this matrix is said to be low rank. Therefore, this paper proposes an HSI recovering algorithm from compressed measurements involving a joint sparse and low rank optimization problem, which seeks to jointly minimize the ℓ2-, ℓ1-, and ℓ*-norm, leading the solution to fit the given projections, and be simultaneously sparse and low rank. Several simulations, along different data sets and optical sensing architectures, show that when the low rank property is included in the inverse problem formulation, the reconstruction quality increases up to four (dB) in terms of peak signal to noise ratio.

Coded aperture design in compressive spectral imaging based on side information.

Coded aperture compressive spectral imagers (CSI) sense a three-dimensional data cube by using two-dimensional projections of the coded and spectrally dispersed input image. Recently, it has been shown that by combining spectral images acquired from a CSI sensor and a complementary sensor leads to substantial improvement in the quality of the fused image. To maximally exploit the benefits of the complementary information, the spatial structure of the coded apertures must be optimized inasmuch as these structures determine the sensing matrix properties and, accordingly, the quality of the reconstructed images. This paper proposes a method to use side information from a red-green-blue sensor to design the coded aperture patterns of a CSI imager, such that more detailed spatial images and wavelength profiles can be reconstructed. The side information is used as the input of an edge detection algorithm to approximate a version of the edges of the spectral images. The coded apertures are designed to follow the spatial structure determined by the estimated spectral edges, such that the high frequencies are promoted, leading to more detailed reconstructed spectral images. Simulations and experimental results indicate that when compared with random coded aperture structures, the designed coded apertures based on side information obtain up to 3 dB improvement in the quality of the reconstructed images.

Spatiotemporal blue noise coded aperture design for multi-shot compressive spectral imaging.

Multi-shot coded aperture snapshot spectral imaging (CASSI) systems capture the spectral information of a scene using a small set of coded focal plane array (FPA) compressive measurements. Compressed sensing (CS) reconstruction algorithms are then used to reconstruct the underlying spectral 3D data cube from an underdetermined system of linear equations. Multiple snapshots result in a less ill-posed inverse problem and improved reconstructions. The only varying components in CASSI are the coded apertures, whose structure is crucial inasmuch as they determine the minimum number of FPA measurements needed for correct image reconstruction and the corresponding attainable quality. Traditionally, the spatial structures of the coded aperture entries are selected at random, leading to suboptimal reconstruction solutions. This work presents an optimal structure design of a set of coded apertures by optimizing the concentration of measure of the multi-shot CASSI sensing matrix and its incoherence with respect to the sparse representation basis. First, the CASSI matrix system representation in terms of the ensemble of random projections is established. Then, the restricted isometry property (RIP) of the CASSI projections is determined as a function of the coded aperture entries. The optimal coded aperture structures are designed under the criterion of satisfying the RIP with high probability, coined spatiotemporal blue noise (BN) coded apertures. Furthermore, an algorithm that implements the BN ensembles is presented. Extensive simulations and a testbed implementation are developed to illustrate the improvements of the BN coded apertures over the traditionally used coded aperture structures, in terms of spectral image reconstruction PSNR and SSIM.