Convolutional neural network for improved event-based Shack-Hartmann wavefront reconstruction.

Shack-Hartmann wavefront sensing is a technique for measuring wavefront aberrations, whose use in adaptive optics relies on fast position tracking of an array of spots. These sensors conventionally use frame-based cameras operating at a fixed sampling rate to report pixel intensities, even though only a fraction of the pixels have signal. Prior in-lab experiments have shown feasibility of event-based cameras for Shack-Hartmann wavefront sensing (SHWFS), asynchronously reporting the spot locations as log intensity changes at a microsecond time scale. In our work, we propose a convolutional neural network (CNN) called event-based wavefront network (EBWFNet) that achieves highly accurate estimation of the spot centroid position in real time. We developed a custom Shack-Hartmann wavefront sensing hardware with a common aperture for the synchronized frame- and event-based cameras so that spot centroid locations computed from the frame-based camera may be used to train/test the event-CNN-based centroid position estimation method in an unsupervised manner. Field testing with this hardware allows us to conclude that the proposed EBWFNet achieves sub-pixel accuracy in real-world scenarios with substantial improvement over the state-of-the-art event-based SHWFS. An ablation study reveals the impact of data processing, CNN components, and training cost function; and an unoptimized MATLAB implementation is shown to run faster than 800 Hz on a single GPU.

Continuously variable Fourier filters fabricated using varying angle glancing angle deposition for chip-scale spectroscopy.

Low-cost spectroscopy has received a great deal of attention in recent years in applications such as food inspection, disease detection, and manufacturing. Current spectroscopic systems rely on multiple optical components, making them mechanically fragile systems. In our previous work, we demonstrated the use of Fourier filtering using thin dielectric films. The sampling effect from the cavity resonances can be used to decompose a signal into its Fourier components. Although the thin films were deposited directly on the face of the detectors, filters of varying thicknesses were needed, which required multiple lithographic processes. To overcome this challenge, in this work, we use a continuously variable filters deposited by a single-step electron-beam evaporation technique. We demonstrate a novel, to our knowledge, method that utilizes the glancing angle deposition technique with a continuously varying angle in order to produce tens of variable Fourier filters in a single deposition run. To prove this technique, we deposit this variable filter on a 38-channel linear detector and show the results from this device.

Framework for improving DoLP and AoLP reconstruction quality in microgrid polarimeters.

In polarimetric imaging, degree and angle of linear polarization (DoLP and AoLP, respectively) are computed from ratios of Stokes parameters. In snapshot imagers, DoLP and AoLP are degraded by inherent mismatches between the spatial bandwidth of the S0, S1, and S2 parameters reconstructed by demosaicking from microgrid polarizer array (MPA)-sampled data. To overcome this, we rigorously show that log-MPA-sampled data approximately decouples DoLP and AoLP from the intensity component (S0) in the spatial Fourier domain. Based on this analysis, we propose an alternative demosaicking strategy aimed at estimating DoLP and AoLP directly from MPA-sampled data. Our method bypasses Stokes parameter estimation, alleviating the spatial bandwidth mismatch problems altogether and reducing computational complexity. We experimentally verify the superior DoLP and AoLP reconstructions of the proposed log-MPA demosaicking compared to the conventional Stokes parameter demosaicking approach in simulation. We simulated the conventional 2 × 2 MPA patterns as well as the more recently introduced 2 × 4 MPA patterns, and report quantitative results (mean squared error, structural similarity index, and polarization angular error) using five demosaicking approaches drawn from the literature. We also provide a closed-form error analysis on the log-MPA-sampled data to demonstrate that the approximation error is negligible for real practical applications.

Tunable optical filter using phase change materials for smart IR night vision applications.

In this paper we present a tunable filter using Ge2Sb2Se4Te1 (GSST) phase change material. The design principle of the filter is based on a metal-insulator-metal (MIM) cavity operating in the reflection mode. This is intended for night vision applications that utilize 850nm as the illumination source. The filter allows us to selectively reject the 850nm band in one state. This is illustrated through several daytime and nighttime imaging applications.

L's approach: illumination pattern estimation technique for optical microscopy.

We propose a novel solution to the correction of illumination nonuniformity without removing the imaging sample. Calibration of the spatial illumination pattern in reflectance microscopy is challenging due to the fact that the illumination source is colocated with the objective lens and therefore cannot be observed directly. Our proposed methodology overcomes this by collecting three spatially translated images in a strategic way. We prove that 'log-illumination pattern' recovery can be reformulated as a deconvolution of log-ratio of captured images, and develop an efficient, noise-robust implementation. Experiments with simulated and reflectance microscopy data verify the effectiveness of our proposed approach. LAY DESCRIPTION: The proposed technique overcomes a common problem of nonuniform illumination in microscopy. For the case of reflectance microscopy, we show that a few images with large amounts of overlap can be used to recover the illumination pattern. In conjunction with an image formation model, the recovered illumination pattern may be used to correct the images captured under similar illumination conditions.

Fourier multispectral imaging.

Current multispectral imaging systems use narrowband filters to capture the spectral content of a scene, which necessitates different filters to be designed for each application. In this paper, we demonstrate the concept of Fourier multispectral imaging which uses filters with sinusoidally varying transmittance. We designed and built these filters employing a single-cavity resonance, and made spectral measurements with a multispectral LED array. The measurements show that spectral features such as transmission and absorption peaks are preserved with this technique, which makes it a versatile technique than narrowband filters for a wide range of multispectral imaging applications.

Improved microgrid arrangement for integrated imaging polarimeters.

For almost 20 years, microgrid polarimetric imaging systems have been built using a 2×2 repeating pattern of polarization analyzers. In this Letter, we show that superior spatial resolution is achieved over this 2×2 case when the analyzers are arranged in a 2×4 repeating pattern. This unconventional result, in which a more distributed sampling pattern results in finer spatial resolution, is also achieved without affecting the conditioning of the polarimetric data-reduction matrix. Proof is provided theoretically and through Stokes image reconstruction of synthesized data.

Time-Ordered Recent Event (TORE) Volumes for Event Cameras.

Event cameras are an exciting, new sensor modality enabling high-speed imaging with extremely low-latency and wide dynamic range. Unfortunately, most machine learning architectures are not designed to directly handle sparse data, like that generated from event cameras. Many state-of-the-art algorithms for event cameras rely on interpolated event representations-obscuring crucial timing information, increasing the data volume, and limiting overall network performance. This paper details an event representation called Time-Ordered Recent Event (TORE) volumes. TORE volumes are designed to compactly store raw spike timing information with minimal information loss. This bio-inspired design is memory efficient, computationally fast, avoids time-blocking (i.e., fixed and predefined frame rates), and contains "local memory" from past data. The design is evaluated on a wide range of challenging tasks (e.g., event denoising, image reconstruction, classification, and human pose estimation) and is shown to dramatically improve state-of-the-art performance. TORE volumes are an easy-to-implement replacement for any algorithm currently utilizing event representations.

Lossless White Balance for Improved Lossless CFA Image and Video Compression.

Color filter array is a spatial multiplexing of pixel-sized filters fabricated over pixel sensors in most color image sensors. The state-of-the-art lossless coding techniques of raw sensor data captured by such sensors leverage spatial or cross-color correlation using lifting schemes. In this paper, we propose a lifting-based lossless white balance algorithm. When applied to the raw sensor data, the spatial bandwidth of the implied chrominance signals decreases. We propose to use this white balance as a pre-processing step to lossless CFA subsampled image/video compression, improving the overall coding efficiency of the raw sensor data.

Multi-Resolution Aitchison Geometry Image Denoising for Low-Light Photography.

In the low-photon imaging regime, noise in the image sensors is dominated by shot noise, best modeled statistically as Poisson distribution. In this work, we show that the Poisson likelihood function is very well matched with the Bayesian estimation of the "difference of log of contrast of pixel intensities." More specifically, our work is rooted in statistical compositional data analysis, whereby we reinterpret the Aitchison geometry as a multi-resolution analysis in the log-pixel domain. We demonstrate that the difference-log-contrast has wavelet-like properties that correspond well with the human visual system, while being robust to illumination variations. We derive a denoising technique based on an approximate conjugate prior for the latent Aitchison variable that gives rise to an explicit minimum mean squared error estimation. The resulting denoising technique preserves image contrast details that are arguably more meaningful to human vision than the pixel intensity values themselves.

Distance Surface for Event-Based Optical Flow.

We propose DistSurf-OF, a novel optical flow method for neuromorphic cameras. Neuromorphic cameras (or event detection cameras) are an emerging sensor modality that makes use of dynamic vision sensors (DVS) to report asynchronously the log-intensity changes (called "events") exceeding a predefined threshold at each pixel. In absence of the intensity value at each pixel location, we introduce a notion of "distance surface"-the distance transform computed from the detected events-as a proxy for object texture. The distance surface is then used as an input to the intensity-based optical flow methods to recover the two dimensional pixel motion. Real sensor experiments verify that the proposed DistSurf-OF accurately estimates the angle and speed of each events.

Corrupted Reference Image Quality Assessment of Denoised Images.

We propose corrupted reference image quality assessment (CRIQA), a novel foundation for reasoning about image quality and image denoising problems jointly. In order to assess the visual quality of a processed image relative to an ideal reference image (not provided), we predict the full-reference image quality assessment (FRIQA) scores of denoised images without having the direct access to the ideal reference image, but with the help of the observed corrupted image, instead. Our simulation studies verify that the CRIQA scores of denoised images indeed agree with the corresponding FRIQA scores, and human subject studies confirm that CRIQA scores are more consistent with the perceived image denoising quality than the NRIQA scores. We demonstrated the usefulness of CRIQA with an application in denoising parameter tuning.

Spatio-spectral color filter array design for optimal image recovery.

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array-a physical construction whereby only a single color value is measured at each pixel location. Owing to the growing ubiquity of color imaging and display devices, much recent work has focused on the implications of such arrays for subsequent digital processing, including in particular the canonical demosaicking task of reconstructing a full color image from spatially subsampled and incomplete color data acquired under a particular choice of array pattern. In contrast to the majority of the demosaicking literature, we consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem formally as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, and-after proving sub-optimality of a wide class of existing array patterns-provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple color image test sets support our theoretical results, and indicate the potential of these patterns to increase spatial resolution for fixed sensor size, and to contribute to improved reconstruction fidelity as well as significantly reduced hardware complexity.

Towards Optimal Denoising of Image Contrast.

Most conventional imaging modalities detect light indirectly by observing high-energy photons. The random nature of photon emission and detection is often the dominant sources of noise in imaging. Such case is referred to as photon-limited imaging, and the noise distribution is well modeled as Poisson. Multiplicative multiscale innovation (MMI) presents a natural model for Poisson count measurement, where the inter-scale relation is represented as random partitioning (binomial distribution) or local image contrast. In this paper, we propose a nonparametric empirical Bayes estimator that minimizes the mean square error of MMI coefficients. The proposed method achieves better performance compared with state-of-the-art methods in both synthetic and real sensor image experiments under low illumination.

Monte-Carlo Acceleration of Bilateral Filter and Non-Local Means.

We propose stochastic bilateral filter (SBF) and stochastic non-local means (SNLM), efficient randomized processes that agree with conventional bilateral filter (BF) and non-local means (NLM) on average, respectively. By Monte-Carlo, we repeat this process a few times with different random instantiations so that they can be averaged to attain the correct BF/NLM output. The computational bottleneck of the SBF and SNLM are constant with respect to the window size and the color dimension of the edge image, meaning the execution times for color and hyperspectral images are nearly the same as for the grayscale images. In addition, for SNLM, the complexity is constant with respect to the block size. The proposed stochastic filter implementations are considerably faster than the conventional and existing "fast" implementations for high dimensional image data.

Camera-Aware Multi-Resolution Analysis for Raw Image Sensor Data Compression.

We propose novel lossless and lossy compression schemes for color filter array (CFA) sampled images based on the Camera-A ware Multi-Resolution Analysis, or CAMRA. Specifically, by CAMRA we refer to modifications that we make to wavelet transform of CFA sampled images in order to achieve a very high degree of decorrelation at the finest scale wavelet coefficients; and a series of color processing steps applied to the coarse scale wavelet coefficients, aimed at limiting the propagation of lossy compression errors through the subsequent camera processing pipeline. We validated our theoretical analysis and the performance of the proposed compression schemes using the images of natural scenes captured in a raw format. The experimental results verify that our proposed methods improve coding efficiency relative to the standard and the state-of-the-art compression schemes for CFA sampled images.

Pixel Binning for High Dynamic Range Color Image Sensor Using Square Sampling Lattice.

We propose a new pixel binning scheme for color image sensors. We minimized distortion caused by binning by requiring that the superpixels lie on a square sampling lattice. The proposed binning schemes achieve the equivalent of 4.42 times signal strength improvement with the image resolution loss of 5 times, higher in noise performance and in resolution than the existing binning schemes. As a result, the proposed binning has considerably less artifacts and better noise performance compared with the existing binning schemes. In addition, we provide an extension to the proposed binning scheme for performing single-shot high dynamic range image acquisition.

Improved Denoising via Poisson Mixture Modeling of Image Sensor Noise.

This paper describes a study aimed at comparing the real image sensor noise distribution to the models of noise often assumed in image denoising designs. A quantile analysis in pixel, wavelet transform, and variance stabilization domains reveal that the tails of Poisson, signal-dependent Gaussian, and Poisson-Gaussian models are too short to capture real sensor noise behavior. A new Poisson mixture noise model is proposed to correct the mismatch of tail behavior. Based on the fact that noise model mismatch results in image denoising that undersmoothes real sensor data, we propose a mixture of Poisson denoising method to remove the denoising artifacts without affecting image details, such as edge and textures. Experiments with real sensor data verify that denoising for real image sensor data is indeed improved by this new technique.

Joint Defogging and Demosaicking.

Image defogging is a technique used extensively for enhancing visual quality of images in bad weather conditions. Even though defogging algorithms have been well studied, defogging performance is degraded by demosaicking artifacts and sensor noise amplification in distant scenes. In order to improve the visual quality of restored images, we propose a novel approach to perform defogging and demosaicking simultaneously. We conclude that better defogging performance with fewer artifacts can be achieved when a defogging algorithm is combined with a demosaicking algorithm simultaneously. We also demonstrate that the proposed joint algorithm has the benefit of suppressing noise amplification in distant scenes. In addition, we validate our theoretical analysis and observations for both synthesized data sets with ground truth fog-free images and natural scene data sets captured in a raw format.

Joint demosaicing and denoising.

The output image of a digital camera is subject to a severe degradation due to noise in the image sensor. This paper proposes a novel technique to combine demosaicing and denoising procedures systematically into a single operation by exploiting their obvious similarities. We first design a filter as if we are optimally estimating a pixel value from a noisy single-color (sensor) image. With additional constraints, we show that the same filter coefficients are appropriate for color filter array interpolation (demosaicing) given noisy sensor data. The proposed technique can combine many existing denoising algorithms with the demosaicing operation. In this paper, a total least squares denoising method is used to demonstrate the concept. The algorithm is tested on color images with pseudorandom noise and on raw sensor data from a real CMOS digital camera that we calibrated. The experimental results confirm that the proposed method suppresses noise (CMOS/CCD image sensor noise model) while effectively interpolating the missing pixel components, demonstrating a significant improvement in image quality when compared to treating demosaicing and denoising problems independently.