Reduced spatiotemporal bandwidth of motion stabilized event-based imaging sensors: experimental demonstration.

This work investigates event-based sensor (EBS) imaging system's read-out bandwidth performance under linear motion, with and without hardware stabilization techniques. We implement three image stabilization methods using hardware rotation to cancel the sensor platform's linear motion and recapture lost EBS performance. We successfully demonstrated the methods, showing a bandwidth reduction of over an order of magnitude in two scenes, 10 scene variations, and five EBS velocities. This work demonstrates the benefits of stabilization with EBS to reduce bandwidth requirements versus unstabilized EBS systems.

Bounds on mutual information of mixture data for classification tasks.

To quantify the optimum performance for classification tasks, the Shannon mutual information is a natural information-theoretic metric, as it is directly related to the probability of error. The data produced by many imaging systems can be modeled by mixture distributions. The mutual information between mixture data and the class label does not have an analytical expression nor any efficient computational algorithms. We introduce a variational upper bound, a lower bound, and three approximations, all employing pair-wise divergences between mixture components. We compare the new bounds and approximations with Monte Carlo stochastic sampling and bounds derived from entropy bounds. To conclude, we evaluate the performance of the bounds and approximations through numerical simulations.

Invertibility of multi-energy X-ray transform.

PURPOSE: The goal is to provide a sufficient condition for the invertibility of a multi-energy (ME) X-ray transform. The energy-dependent X-ray attenuation profiles can be represented by a set of coefficients using the Alvarez-Macovski (AM) method. An ME X-ray transform is a mapping from N AM coefficients to N noise-free energy-weighted measurements, where N ≥ 2 . METHODS: We apply a general invertibility theorem to prove the equivalence of global and local invertibility for an ME X-ray transform. We explore the global invertibility through testing whether the Jacobian of the mapping J ( A ) has zero values over the support of the mapping. The Jacobian of an arbitrary ME X-ray transform is an integration over all spectral measurements. A sufficient condition for J ( A ) ≠ 0 for all A is that the integrand of J ( A ) is ≥ 0 (or ≤ 0 ) everywhere. Note that the trivial case of the integrand equals 0 everywhere is ignored. Using symmetry, we simplified the integrand of the Jacobian to three factors that are determined by the total attenuation, the basis functions, and the energy-weighting functions, respectively. The factor related to the total attenuation is always positive; hence, the invertibility of the X-ray transform can be determined by testing the signs of the other two factors. Furthermore, we use the Cramér-Rao lower bound (CRLB) to characterize the noise-induced estimation uncertainty and provide a maximum-likelihood (ML) estimator. RESULTS: The factor related to the basis functions is always negative when the photoelectric/Compton/Rayleigh basis functions are used and K-edge materials are not considered. The sign of the energy-weighting factor depends on the system source spectra and the detector response functions. For four special types of X-ray detectors, the sign of this factor stays the same over the integration range. Therefore, when these four types of detectors are used for imaging non-K-edge materials, the ME X-ray transform is globally invertible. The same framework can be used to study an arbitrary ME X-ray imaging system, for example, when K-edge materials are present. Furthermore, the ML estimator we presented is an unbiased, efficient estimator and can be used for a wide range of scenes. CONCLUSIONS: We have provided a framework to study the invertibility of an arbitrary ME X-ray transform and proved the global invertibility for four types of systems.

Approaching quantum-limited imaging resolution without prior knowledge of the object location.

Passive imaging receivers that demultiplex an incoherent optical field into a set of orthogonal spatial modes prior to detection can surpass canonical diffraction limits on spatial resolution. However, these mode-sorting receivers exhibit sensitivity to contextual nuisance parameters (e.g., the centroid of a clustered or extended object), raising questions on their viability in realistic scenarios where prior information about the scene is limited. We propose a multistage detection strategy that segments the total recording time between different physical measurements to build up the required prior information for near quantum-optimal imaging performance at sub-Rayleigh length scales. We show, via Monte Carlo simulations, that an adaptive two-stage scheme that dynamically allocates recording time between a conventional direct detection measurement and a binary mode sorter outperforms idealized direct detection alone when no prior knowledge of the object centroid is available, achieving one to two orders of magnitude improvement in mean squared error for simple estimation tasks. Our scheme can be generalized for more sophisticated tasks involving multiple parameters and/or minimal prior information.

Visibility of Quantization Errors in Reversible JPEG2000.

Image compression systems that exploit the properties of the human visual system have been studied extensively over the past few decades. For the JPEG2000 image compression standard, all previous methods that aim to optimize perceptual quality have considered the irreversible pipeline of the standard. In this work, we propose an approach for the reversible pipeline of the JPEG2000 standard. We introduce a new methodology to measure visibility of quantization errors when reversible color and wavelet transforms are employed. Incorporation of the visibility thresholds using this methodology into a JPEG2000 encoder enables creation of scalable codestreams that can provide both near-threshold and numerically lossless representations, which is desirable in applications where restoration of original image samples is required. Most importantly, this is the first work that quantifies the bitrate penalty incurred by the reversible transforms in near-threshold image compression compared to the irreversible transforms.

Eavesdropping of display devices by measurement of polarized reflected light.

Display devices, or displays, such as those utilized extensively in cell phones, computer monitors, televisions, instrument panels, and electronic signs, are polarized light sources. Most displays are designed for direct viewing by human eyes, but polarization imaging of reflected light from a display can also provide valuable information. These indirect (reflected/scattered) photons, which are often not in direct field-of-view and mixed with photons from the ambient light, can be extracted to infer information about the content on the display devices. In this work, we apply Stokes algebra and Mueller calculus with the edge overlap technique to the problem of extracting indirect photons reflected/scattered from displays. Our method applies to recovering information from linearly and elliptically polarized displays that are reflected by transmissive surfaces, such as glass, and semi-diffuse opaque surfaces, such as marble tiles and wood furniture. The technique can further be improved by applying Wiener filtering.

Face recognition with non-greedy information-optimal adaptive compressive imaging.

Adaptive compressive measurements can offer significant system performance advantages due to online learning over non-adaptive or static compressive measurements for a variety of applications, such as image formation and target identification. However, such adaptive measurements tend to be sub-optimal due to their greedy design. Here, we propose a non-greedy adaptive compressive measurement design framework and analyze its performance for a face recognition task. While a greedy adaptive design aims to optimize the system performance on the next immediate measurement, a non-greedy adaptive design goes beyond that by strategically maximizing the system performance over all future measurements. Our non-greedy adaptive design pursues a joint optimization of measurement design and photon allocation within a rigorous information-theoretic framework. For a face recognition task, simulation studies demonstrate that the proposed non-greedy adaptive design achieves a nearly two to three fold lower probability of misclassification relative to the greedy adaptive and static designs. The simulation results are validated experimentally on a compressive optical imager testbed.

Single molecule image formation, reconstruction and processing: introduction.

The ability to image at the single molecule scale has revolutionized research in molecular biology. This feature issue presents a collection of articles that provides new insights into the fundamental limits of single molecule imaging and reports novel techniques for image formation and analysis.

Capacity of electromagnetic communication modes in a noise-limited optical system.

We present capacity bounds of an optical system that communicates using electromagnetic waves between a transmitter and a receiver. The bounds are investigated in conjunction with a rigorous theory of degrees of freedom (DOF) in the presence of noise. By taking into account the different signal-to-noise ratio (SNR) levels, an optimal number of DOF that provides the maximum capacity is defined. We find that for moderate noise levels, the DOF estimate of the number of active modes is approximately equal to the optimum number of channels obtained by a more rigorous water-filling procedure. On the other hand, for very low- or high-SNR regions, the maximum capacity can be obtained using less or more channels compared to the number of communicating modes given by the DOF theory. In general, the capacity is shown to increase with increasing size of the transmitting and receiving volumes, whereas it decreases with an increase in the separation between volumes. Under the practical channel constraints of noise and finite available power, the capacity upper bound can be estimated by the well-known iterative water-filling solution to determine the optimal power allocation into the subchannels corresponding to the set of singular values when channel state information is known at the transmitter.

Cellular imaging of deep organ using two-photon Bessel light-sheet nonlinear structured illumination microscopy.

In vivo fluorescent cellular imaging of deep internal organs is highly challenging, because the excitation needs to penetrate through strong scattering tissue and the emission signal is degraded significantly by photon diffusion induced by tissue-scattering. We report that by combining two-photon Bessel light-sheet microscopy with nonlinear structured illumination microscopy (SIM), live samples up to 600 microns wide can be imaged by light-sheet microscopy with 500 microns penetration depth, and diffused background in deep tissue light-sheet imaging can be reduced to obtain clear images at cellular resolution in depth beyond 200 microns. We demonstrate in vivo two-color imaging of pronephric glomeruli and vasculature of zebrafish kidney, whose cellular structures located at the center of the fish body are revealed in high clarity by two-color two-photon Bessel light-sheet SIM.

Information optimal compressive sensing: static measurement design.

The compressive sensing paradigm exploits the inherent sparsity/compressibility of signals to reduce the number of measurements required for reliable reconstruction/recovery. In many applications additional prior information beyond signal sparsity, such as structure in sparsity, is available, and current efforts are mainly limited to exploiting that information exclusively in the signal reconstruction problem. In this work, we describe an information-theoretic framework that incorporates the additional prior information as well as appropriate measurement constraints in the design of compressive measurements. Using a Gaussian binomial mixture prior we design and analyze the performance of optimized projections relative to random projections under two specific design constraints and different operating measurement signal-to-noise ratio (SNR) regimes. We find that the information-optimized designs yield significant, in some cases nearly an order of magnitude, improvements in the reconstruction performance with respect to the random projections. These improvements are especially notable in the low measurement SNR regime where the energy-efficient design of optimized projections is most advantageous. In such cases, the optimized projection design departs significantly from random projections in terms of their incoherence with the representation basis. In fact, we find that the maximizing incoherence of projections with the representation basis is not necessarily optimal in the presence of additional prior information and finite measurement noise/error. We also apply the information-optimized projections to the compressive image formation problem for natural scenes, and the improved visual quality of reconstructed images with respect to random projections and other compressive measurement design affirms the overall effectiveness of the information-theoretic design framework.

Space-time compressive imaging.

Compressive imaging systems typically exploit the spatial correlation of the scene to facilitate a lower dimensional measurement relative to a conventional imaging system. In natural time-varying scenes there is a high degree of temporal correlation that may also be exploited to further reduce the number of measurements. In this work we analyze space-time compressive imaging using Karhunen-Loève (KL) projections for the read-noise-limited measurement case. Based on a comprehensive simulation study, we show that a KL-based space-time compressive imager offers higher compression relative to space-only compressive imaging. For a relative noise strength of 10% and reconstruction error of 10%, we find that space-time compressive imaging with 8×8×16 spatiotemporal blocks yields about 292× compression compared to a conventional imager, while space-only compressive imaging provides only 32× compression. Additionally, under high read-noise conditions, a space-time compressive imaging system yields lower reconstruction error than a conventional imaging system due to the multiplexing advantage. We also discuss three electro-optic space-time compressive imaging architecture classes, including charge-domain processing by a smart focal plane array (FPA). Space-time compressive imaging using a smart FPA provides an alternative method to capture the nonredundant portions of time-varying scenes.

Compressive imaging: hybrid measurement basis design.

The inherent redundancy in natural scenes forms the basis of compressive imaging where the number of measurements is less than the dimensionality of the scene. The compressed sensing theory has shown that a purely random measurement basis can yield good reconstructions of sparse objects with relatively few measurements. However, additional prior knowledge about object statistics that is typically available is not exploited in the design of the random basis. In this work, we describe a hybrid measurement basis design that exploits the power spectral density statistics of natural scenes to minimize the reconstruction error by employing an optimal combination of a nonrandom basis and a purely random basis. Using simulation studies, we quantify the reconstruction error improvement achievable with the hybrid basis for a diverse set of natural images. We find that the hybrid basis can reduce the reconstruction error up to 77% or equivalently requires fewer measurements to achieve a desired reconstruction error compared to the purely random basis. It is also robust to varying levels of object sparsity and yields as much as 40% lower reconstruction error compared to the random basis in the presence of measurement noise.

Object reconstruction from adaptive compressive measurements in feature-specific imaging.

Static feature-specific imaging (SFSI), where the measurement basis remains fixed/static during the data measurement process, has been shown to be superior to conventional imaging for reconstruction tasks. Here, we describe an adaptive approach that utilizes past measurements to inform the choice of measurement basis for future measurements in an FSI system, with the goal of maximizing the reconstruction fidelity while employing the fewest measurements. An algorithm to implement this adaptive approach is developed for FSI systems, and the resulting systems are referred to as adaptive FSI (AFSI) systems. A simulation study is used to analyze the performance of the AFSI system for two choices of measurement basis: principal component (PC) and Hadamard. Here, the root mean squared error (RMSE) metric is employed to quantify the reconstruction fidelity. We observe that an AFSI system achieves as much as 30% lower RMSE compared to an SFSI system. The performance improvement of the AFSI systems is verified using an experimental setup employed using a digital micromirror device (DMD) array.

Increased field of view through optical multiplexing.

Traditional approaches to wide field of view (FoV) imager design usually lead to overly complex optics with high optical mass and/or pan-tilt mechanisms that incur significant mechanical/weight penalties, which limit their applications, especially on mobile platforms such as unmanned aerial vehicles (UAVs).We describe a compact wide FoV imager design based on superposition imaging that employs thin film shutters and multiple beamsplitters to reduce system weight and eliminate mechanical pointing. The performance of the superposition wide FoV imager is quantified using a simulation study and is experimentally demonstrated. Here, a threefold increase in the FoV relative to the narrow FoV imaging optics employed imager design is realized. The performance of a superposition wide FoV imager is analyzed relative to a traditional wide FoV imager and we find that it can offer comparable performance.

Point spread function engineering for iris recognition system design.

Undersampling in the detector array degrades the performance of iris-recognition imaging systems. We find that an undersampling of 8 x 8 reduces the iris-recognition performance by nearly a factor of 4 (on CASIA iris database), as measured by the false rejection ratio (FRR) metric. We employ optical point spread function (PSF) engineering via a Zernike phase mask in conjunction with multiple subpixel shifted image measurements (frames) to mitigate the effect of undersampling. A task-specific optimization framework is used to engineer the optical PSF and optimize the postprocessing parameters to minimize the FRR. The optimized Zernike phase enhanced lens (ZPEL) imager design with one frame yields an improvement of nearly 33% relative to a thin observation module by bounded optics (TOMBO) imager with one frame. With four frames the optimized ZPEL imager achieves a FRR equal to that of the conventional imager without undersampling. Further, the ZPEL imager design using 16 frames yields a FRR that is actually 15% lower than that obtained with the conventional imager without undersampling.

Compressive imaging system design using task-specific information.

We present a task-specific information (TSI) based framework for designing compressive imaging (CI) systems. The task of target detection is chosen to demonstrate the performance of the optimized CI system designs relative to a conventional imager. In our optimization framework, we first select a projection basis and then find the associated optimal photon-allocation vector in the presence of a total photon-count constraint. Several projection bases, including principal components (PC), independent components, generalized matched-filter, and generalized Fisher discriminant (GFD) are considered for candidate CI systems, and their respective performance is analyzed for the target-detection task. We find that the TSI-optimized CI system design based on a GFD projection basis outperforms all other candidate CI system designs as well as the conventional imager. The GFD-based compressive imager yields a TSI of 0.9841 bits (out of a maximum possible 1 bit for the detection task), which is nearly ten times the 0.0979 bits achieved by the conventional imager at a signal-to-noise ratio of 5.0. We also discuss the relation between the information-theoretic TSI metric and a conventional statistical metric like probability of error in the context of the target-detection problem. It is shown that the TSI can be used to derive an upper bound on the probability of error that can be attained by any detection algorithm.

Task-specific information for imaging system analysis.

Imagery is often used to accomplish some computational task. In such cases there are some aspects of the imagery that are relevant to the task and other aspects that are not. In order to quantify the task-specific quality of such imagery, we introduce the concept of task-specific information (TSI). A formal framework for the computation of TSI is described and is applied to three common tasks: target detection, classification, and localization. We demonstrate the utility of TSI as a metric for evaluating the performance of three imaging systems: ideal geometric, diffraction-limited, and projective. The TSI results obtained from the simulation study quantify the degradation in the task-specific performance with optical blur. We also demonstrate that projective imagers can provide higher TSI than conventional imagers at small signal-to-noise ratios.

Pseudorandom phase masks for superresolution imaging from subpixel shifting.

We present a method for overcoming the pixel-limited resolution of digital imagers. Our method combines optical point-spread function engineering with subpixel image shifting. We place an optimized pseudorandom phase mask in the aperture stop of a conventional imager and demonstrate the improved performance that can be achieved by combining multiple subpixel shifted images. Simulation results show that the pseudorandom phase-enhanced lens (PRPEL) imager achieves as much as 50% resolution improvement over a conventional multiframe imager. The PRPEL imager also enhances reconstruction root-mean-squared error by as much as 20%. We present experimental results that validate the predicted PRPEL imager performance.

Information-based analysis of simple incoherent imaging systems.

We present an information-based analysis of three candidate imagers: a conventional lens system, a cubic phase mask system, and a random phase mask system. For source volumes comprising relatively few equal-intensity point sources we compare both the axial and lateral information content of detector intensity measurements. We include the effect of additive white Gaussian noise. Single and distributed aperture imaging is studied. A single detector in each of two apertures using conventional lenses can yield 36% of the available scene information when the source volume contains only single point source. The addition of cubic phase masks yields nearly 74% of the scene information. An identical configuration using random phase masks offers the best performance with 89% scene information available in the detector intensity measurements.