Shedding light on 19th century spectra by analyzing Lippmann photography.

From uncovering the structure of the atom to the nature of the universe, spectral measurements have helped some of science's greatest discoveries. While pointwise spectral measurements date back to Newton, it is commonly thought that hyperspectral images originated in the 1970s. However, the first hyperspectral images are over a century old and are locked in the safes of a handful of museums. These hidden treasures are examples of the first color photographs and earned their inventor, Gabriel Lippmann, the 1908 Nobel Prize in Physics. Since the original work of Lippmann, the process has been predominately understood from the monochromatic perspective, with analogies drawn to Bragg gratings, and the polychromatic case treated as a simple extension. As a consequence, there are misconceptions about the invertibility of the Lippmann process. We show that the multispectral image reflected from a Lippmann plate contains distortions that are not explained by current models. We describe these distortions by directly modeling the process for general spectra and devise an algorithm to recover the original spectra. This results in a complete analysis of the Lippmann process. Finally, we demonstrate the accuracy of our recovery algorithm on self-made Lippmann plates, for which the acquisition setup is fully understood. However, we show that, in the case of historical plates, there are too many unknowns to reliably recover 19th century spectra of natural scenes.

Learning neural connectivity from firing activity: efficient algorithms with provable guarantees on topology.

The connectivity of a neuronal network has a major effect on its functionality and role. It is generally believed that the complex network structure of the brain provides a physiological basis for information processing. Therefore, identifying the network's topology has received a lot of attentions in neuroscience and has been the center of many research initiatives such as Human Connectome Project. Nevertheless, direct and invasive approaches that slice and observe the neural tissue have proven to be time consuming, complex and costly. As a result, the inverse methods that utilize firing activity of neurons in order to identify the (functional) connections have gained momentum recently, especially in light of rapid advances in recording technologies; It will soon be possible to simultaneously monitor the activities of tens of thousands of neurons in real time. While there are a number of excellent approaches that aim to identify the functional connections from firing activities, the scalability of the proposed techniques plays a major challenge in applying them on large-scale datasets of recorded firing activities. In exceptional cases where scalability has not been an issue, the theoretical performance guarantees are usually limited to a specific family of neurons or the type of firing activities. In this paper, we formulate the neural network reconstruction as an instance of a graph learning problem, where we observe the behavior of nodes/neurons (i.e., firing activities) and aim to find the links/connections. We develop a scalable learning mechanism and derive the conditions under which the estimated graph for a network of Leaky Integrate and Fire (LIf) neurons matches the true underlying synaptic connections. We then validate the performance of the algorithm using artificially generated data (for benchmarking) and real data recorded from multiple hippocampal areas in rats.

Acoustic echoes reveal room shape.

Imagine that you are blindfolded inside an unknown room. You snap your fingers and listen to the room's response. Can you hear the shape of the room? Some people can do it naturally, but can we design computer algorithms that hear rooms? We show how to compute the shape of a convex polyhedral room from its response to a known sound, recorded by a few microphones. Geometric relationships between the arrival times of echoes enable us to "blindfoldedly" estimate the room geometry. This is achieved by exploiting the properties of Euclidean distance matrices. Furthermore, we show that under mild conditions, first-order echoes provide a unique description of convex polyhedral rooms. Our algorithm starts from the recorded impulse responses and proceeds by learning the correct assignment of echoes to walls. In contrast to earlier methods, the proposed algorithm reconstructs the full 3D geometry of the room from a single sound emission, and with an arbitrary geometry of the microphone array. As long as the microphones can hear the echoes, we can position them as we want. Besides answering a basic question about the inverse problem of room acoustics, our results find applications in areas such as architectural acoustics, indoor localization, virtual reality, and audio forensics.

Locating the source of diffusion in large-scale networks.

How can we localize the source of diffusion in a complex network? Because of the tremendous size of many real networks-such as the internet or the human social graph-it is usually unfeasible to observe the state of all nodes in a network. We show that it is fundamentally possible to estimate the location of the source from measurements collected by sparsely placed observers. We present a strategy that is optimal for arbitrary trees, achieving maximum probability of correct localization. We describe efficient implementations with complexity O(N(α)), where α=1 for arbitrary trees and α=3 for arbitrary graphs. In the context of several case studies, we determine how localization accuracy is affected by various system parameters, including the structure of the network, the density of observers, and the number of observed cascades.

Three-Dimensional Cubic Barcodes.

We consider three-dimensional cubic barcodes, consisting of smaller cubes, each built from one of two possible materials and carry one bit of information. To retrieve the information stored in the barcode, we measure a 2D projection of the barcode using a penetrating wave such as X-rays, either using parallel-beam or cone-beam scanners from an unknown direction. We derive a theoretical representation of this scanning process and show that for a known barcode pose with respect to the scanner, the projection operator is linear and can be easily inverted. Moreover, we provide a method to estimate the unknown pose of the barcode from a single 2D scan. We also propose coding schemes to correct errors and ambiguities in the reconstruction process. Finally, we test our designed barcode and reconstruction algorithms with several simulations, as well as a real-world barcode acquired with an X-ray cone-beam scanner, as a proof of concept.

Bound and Conquer: Improving Triangulation by Enforcing Consistency.

We study the accuracy of triangulation in multi-camera systems with respect to the number of cameras. We show that, under certain conditions, the optimal achievable reconstruction error decays quadratically as more cameras are added to the system. Furthermore, we analyze the error decay-rate of major state-of-the-art algorithms with respect to the number of cameras. To this end, we introduce the notion of consistency for triangulation, and show that consistent reconstruction algorithms achieve the optimal quadratic decay, which is asymptotically faster than some other methods. Finally, we present simulations results supporting our findings. Our simulations have been implemented in MATLAB and the resulting code is available in the supplementary material, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TPAMI.2019.2939530.

Space-frequency quantization for image compression with directionlets.

The standard separable 2-D wavelet transform (WT) has recently achieved a great success in image processing because it provides a sparse representation of smooth images. However, it fails to efficiently capture 1-D discontinuities, like edges or contours. These features, being elongated and characterized by geometrical regularity along different directions, intersect and generate many large magnitude wavelet coefficients. Since contours are very important elements in the visual perception of images, to provide a good visual quality of compressed images, it is fundamental to preserve good reconstruction of these directional features. In our previous work, we proposed a construction of critically sampled perfect reconstruction transforms with directional vanishing moments imposed in the corresponding basis functions along different directions, called directionlets. In this paper, we show how to design and implement a novel efficient space-frequency quantization (SFQ) compression algorithm using directionlets. Our new compression method outperforms the standard SFQ in a rate-distortion sense, both in terms of mean-square error and visual quality, especially in the low-rate compression regime. We also show that our compression method, does not increase the order of computational complexity as compared to the standard SFQ algorithm.

Dynamic measurement of room impulse responses using a moving microphone.

A technique for the recording of large sets of room impulse responses or head-related transfer functions is presented. The technique uses a microphone moving with constant speed. Given a setup (e.g., length of the room impulse response), a careful choice of the recording parameters (excitation signal, speed of movement) leads to the reconstruction of all impulse responses along the trajectory. In the case of a moving microphone along a circle, the maximal angular speed is given as a function of the length of the impulse response, its maximal temporal frequency, the speed of sound propagation, and the radius of the circle. As a result of the presented algorithm, head-related transfer functions sampled at 44.1 kHz can be measured at all angular positions along the horizontal plane in less than 1 s. The presented theory is compared with a real system implementation using a precision moving microphone holder. The practical setup is discussed together with its limitations.

Locally adaptive wavelet-based image interpolation.

We describe a spatially adaptive algorithm for image interpolation. The algorithm uses a wavelet transform to extract information about sharp variations in the low-resolution image and then implicitly applies interpolation which adapts to the image local smoothness/singularity characteristics. The proposed algorithm yields images that are sharper compared to several other methods that we have considered in this paper. Better performance comes at the expense of higher complexity.

Directionlets: anisotropic multidirectional representation with separable filtering.

In spite of the success of the standard wavelet transform (WT) in image processing in recent years, the efficiency of its representation is limited by the spatial isotropy of its basis functions built in the horizontal and vertical directions. One-dimensional (1-D) discontinuities in images (edges and contours) that are very important elements in visual perception, intersect too many wavelet basis functions and lead to a nonsparse representation. To efficiently capture these anisotropic geometrical structures characterized by many more than the horizontal and vertical directions, a more complex multidirectional (M-DIR) and anisotropic transform is required. We present a new lattice-based perfect reconstruction and critically sampled anisotropic M-DIR WT. The transform retains the separable filtering and subsampling and the simplicity of computations and filter design from the standard two-dimensional WT, unlike in the case of some other directional transform constructions (e.g., curvelets, contourlets, or edgelets). The corresponding anisotropic basis unctions (directionlets) have directional vanishing moments along any two directions with rational slopes. Furthermore, we show that this novel transform provides an efficient tool for nonlinear approximation of images, achieving the approximation power O(N(-1.55)), which, while slower than the optimal rate O(N(-2)), is much better than O(N(-1)) achieved with wavelets, but at similar complexity.

Shapes From Pixels.

Continuous-domain visual signals are usually captured as discrete (digital) images. This operation is not invertible in general, in the sense that the continuous-domain signal cannot be exactly reconstructed based on the discrete image, unless it satisfies certain constraints (e.g., bandlimitedness). In this paper, we study the problem of recovering shape images with smooth boundaries from a set of samples. Thus, the reconstructed image is constrained to regenerate the same samples (consistency), as well as forming a shape (bilevel) image. We initially formulate the reconstruction technique by minimizing the shape perimeter over the set of consistent binary shapes. Next, we relax the non-convex shape constraint to transform the problem into minimizing the total variation over consistent non-negative-valued images. We also introduce a requirement (called reducibility) that guarantees equivalence between the two problems. We illustrate that the reducibility property effectively sets a requirement on the minimum sampling density. We also evaluate the performance of the relaxed alternative in various numerical experiments.

VITRAIL: Acquisition, Modeling, and Rendering of Stained Glass.

Stained glass windows are designed to reveal their powerful artistry under diverse and time-varying lighting conditions; virtual relighting of stained glass, therefore, represents an exceptional tool for the appreciation of this age old art form. However, as opposed to most other artifacts, stained glass windows are extremely difficult if not impossible to analyze using controlled illumination because of their size and position. In this paper, we present novel methods built upon image based priors to perform virtual relighting of stained glass artwork by acquiring the actual light transport properties of a given artifact. In a preprocessing step, we build a material-dependent dictionary for light transport by studying the scattering properties of glass samples in a laboratory setup. We can now use the dictionary to recover a light transport matrix in two ways: under controlled illuminations the dictionary constitutes a sparsifying basis for a compressive sensing acquisition, while in the case of uncontrolled illuminations the dictionary is used to perform sparse regularization. The proposed basis preserves volume impurities and we show that the retrieved light transport matrix is heterogeneous, as in the case of real world objects. We present the rendering results of several stained glass artifacts, including the Rose Window of the Cathedral of Lausanne, digitized using the presented methods.

The contourlet transform: an efficient directional multiresolution image representation.

The limitations of commonly used separable extensions of one-dimensional transforms, such as the Fourier and wavelet transforms, in capturing the geometry of image edges are well known. In this paper, we pursue a "true" two-dimensional transform that can capture the intrinsic geometrical structure that is key in visual information. The main challenge in exploring geometry in images comes from the discrete nature of the data. Thus, unlike other approaches, such as curvelets, that first develop a transform in the continuous domain and then discretize for sampled data, our approach starts with a discrete-domain construction and then studies its convergence to an expansion in the continuous domain. Specifically, we construct a discrete-domain multiresolution and multidirection expansion using nonseparable filter banks, in much the same way that wavelets were derived from filter banks. This construction results in a flexible multiresolution, local, and directional image expansion using contour segments, and, thus, it is named the contourlet transform. The discrete contourlet transform has a fast iterated filter bank algorithm that requires an order N operations for N-pixel images. Furthermore, we establish a precise link between the developed filter bank and the associated continuous-domain contourlet expansion via a directional multiresolution analysis framework. We show that with parabolic scaling and sufficient directional vanishing moments, contourlets achieve the optimal approximation rate for piecewise smooth functions with discontinuities along twice continuously differentiable curves. Finally, we show some numerical experiments demonstrating the potential of contourlets in several image processing applications. Index Terms-Contourlets, contours, filter banks, geometric image processing, multidirection, multiresolution, sparse representation, wavelets.

Rate-distortion optimized tree-structured compression algorithms for piecewise polynomial images.

This paper presents novel coding algorithms based on tree-structured segmentation, which achieve the correct asymptotic rate-distortion (R-D) behavior for a simple class of signals, known as piecewise polynomials, by using an R-D based prune and join scheme. For the one-dimensional case, our scheme is based on binary-tree segmentation of the signal. This scheme approximates the signal segments using polynomial models and utilizes an R-D optimal bit allocation strategy among the different signal segments. The scheme further encodes similar neighbors jointly to achieve the correct exponentially decaying R-D behavior (D(R) - c(o)2(-c1R)), thus improving over classic wavelet schemes. We also prove that the computational complexity of the scheme is of O(N log N). We then show the extension of this scheme to the two-dimensional case using a quadtree. This quadtree-coding scheme also achieves an exponentially decaying R-D behavior, for the polygonal image model composed of a white polygon-shaped object against a uniform black background, with low computational cost of O(N log N). Again, the key is an R-D optimized prune and join strategy. Finally, we conclude with numerical results, which show that the proposed quadtree-coding scheme outperforms JPEG2000 by about 1 dB for real images, like cameraman, at low rates of around 0.15 bpp.

Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance.

We present a statistical view of the texture retrieval problem by combining the two related tasks, namely feature extraction (FE) and similarity measurement (SM), into a joint modeling and classification scheme. We show that using a consistent estimator of texture model parameters for the FE step followed by computing the Kullback-Leibler distance (KLD) between estimated models for the SM step is asymptotically optimal in term of retrieval error probability. The statistical scheme leads to a new wavelet-based texture retrieval method that is based on the accurate modeling of the marginal distribution of wavelet coefficients using generalized Gaussian density (GGD) and on the existence a closed form for the KLD between GGDs. The proposed method provides greater accuracy and flexibility in capturing texture information, while its simplified form has a close resemblance with the existing methods which uses energy distribution in the frequency domain to identify textures. Experimental results on a database of 640 texture images indicate that the new method significantly improves retrieval rates, e.g., from 65% to 77%, compared with traditional approaches, while it retains comparable levels of computational complexity.

The finite ridgelet transform for image representation.

The ridgelet transform was introduced as a sparse expansion for functions on continuous spaces that are smooth away from discontinuities along lines. We propose an orthonormal version of the ridgelet transform for discrete and finite-size images. Our construction uses the finite Radon transform (FRAT) as a building block. To overcome the periodization effect of a finite transform, we introduce a novel ordering of the FRAT coefficients. We also analyze the FRAT as a frame operator and derive the exact frame bounds. The resulting finite ridgelet transform (FRIT) is invertible, nonredundant and computed via fast algorithms. Furthermore, this construction leads to a family of directional and orthonormal bases for images. Numerical results show that the FRIT is more effective than the wavelet transform in approximating and denoising images with straight edges.

Distributed successive refinement of multiview images using broadcast advantage.

In environmental monitoring applications, having multiple cameras focus on common scenery increases robustness of the system. To save energy based on user demand, successive refinement image coding is important, as it allows us to progressively request better image quality. By exploiting the broadcast nature and correlation between multiview images, we investigate a two-camera setup and propose a novel two-encoder successive refinement scheme which imitates a ping-pong game. For the bivariate Gaussian case, we prove that this scheme is successively refinable on the theoretical rate-distortion limit of distributed coding (Wagner surface) under arbitrary settings. For stereo-view images, we develop a practical successive refinement coding algorithm using the same idea. The simulation results show that this scheme operates close to the distributed coding bound.

On the bandwidth of the plenoptic function.

The plenoptic function (POF) provides a powerful conceptual tool for describing a number of problems in image/video processing, vision, and graphics. For example, image-based rendering is shown as sampling and interpolation of the POF. In such applications, it is important to characterize the bandwidth of the POF. We study a simple but representative model of the scene where band-limited signals (e.g., texture images) are "painted" on smooth surfaces (e.g., of objects or walls). We show that, in general, the POF is not band limited unless the surfaces are flat. We then derive simple rules to estimate the essential bandwidth of the POF for this model. Our analysis reveals that, in addition to the maximum and minimum depths and the maximum frequency of painted signals, the bandwidth of the POF also depends on the maximum surface slope. With a unifying formalism based on multidimensional signal processing, we can verify several key results in POF processing, such as induced filtering in space and depth-corrected interpolation, and quantify the necessary sampling rates.

Bits from photons: oversampled image acquisition using binary Poisson statistics.

We study a new image sensor that is reminiscent of a traditional photographic film. Each pixel in the sensor has a binary response, giving only a 1-bit quantized measurement of the local light intensity. To analyze its performance, we formulate the oversampled binary sensing scheme as a parameter estimation problem based on quantized Poisson statistics. We show that, with a single-photon quantization threshold and large oversampling factors, the Cramér-Rao lower bound (CRLB) of the estimation variance approaches that of an ideal unquantized sensor, i.e., as if there were no quantization in the sensor measurements. Furthermore, the CRLB is shown to be asymptotically achievable by the maximum-likelihood estimator (MLE). By showing that the log-likelihood function of our problem is concave, we guarantee the global optimality of iterative algorithms in finding the MLE. Numerical results on both synthetic data and images taken by a prototype sensor verify our theoretical analysis and demonstrate the effectiveness of our image reconstruction algorithm. They also suggest the potential application of the oversampled binary sensing scheme in high dynamic range photography.