GAFFE: a gaze-attentive fixation finding engine.

The ability to automatically detect visually interesting regions in images has many practical applications, especially in the design of active machine vision and automatic visual surveillance systems. Analysis of the statistics of image features at observers' gaze can provide insights into the mechanisms of fixation selection in humans. Using a foveated analysis framework, we studied the statistics of four low-level local image features: luminance, contrast, and bandpass outputs of both luminance and contrast, and discovered that image patches around human fixations had, on average, higher values of each of these features than image patches selected at random. Contrast-bandpass showed the greatest difference between human and random fixations, followed by luminance-bandpass, RMS contrast, and luminance. Using these measurements, we present a new algorithm that selects image regions as likely candidates for fixation. These regions are shown to correlate well with fixations recorded from human observers.

Automatic segmentation of inorganic nanoparticles in BF TEM micrographs.

Transmission electron microscopy (TEM) represents a unique and powerful modality for capturing spatial features of nanoparticles, such as size and shape. However, poor statistics arise as a key obstacle, due to the challenge in accurately and automatically segmenting nanoparticles in TEM micrographs. Towards remedying this deficit, we introduce an automatic particle picking device that is based on the concept of variance hybridized mean local thresholding. Validation of this new segmentation model is accomplished by applying a program written in Matlab to a database of 150 bright field TEM micrographs containing approximately 2,000 nanoparticles. We compare the results to global thresholding, local thresholding, and manual segmentation. It is found that this novel automatic particle picking device reduces false positives and false negatives significantly, while increasing the number of individual particles picked on regions of particle overlap.

Angiogenesis in cultured and cryopreserved pancreatic islet grafts.

BACKGROUND: The ability of rat pancreatic islets to revascularize after transplantation was examined via in vitro and in vivo imaging of the microvasculature using laser scanning confocal microscopy (LSCM). METHODS: Cultured or cryoprocessed islets were transplanted at the renal subcapsular site in rats. At various time intervals after transplantation, three-dimensional imaging of the graft was performed by LSCM. In vitro studies were conducted via microvascular corrosion casting of the grafted kidney in situations where it was difficult to obtain in vivo confocal data due to surgical complications. The vascular morphology of the islet grafts was evaluated quantitatively via digital image analysis algorithms to determine the morphology of the neovascular ingrowth and the rate of revascularization. RESULTS: In cultured islet grafts, the initiation of angiogenesis was observed within 1 week, characterized by the presence of capillary sprouts, tortuous vessels, and blood vessels with blind ends. The revascularization of the graft was typically completed within 2 weeks and could be distinguished as a network of completely perfused blood vessels consisting of intertwining capillaries, with surrounding arterioles and venules. The angiogenesis process in cryopreserved islet grafts required a longer time period to initiate (approximately 2 weeks), and the revascularization was completed in 1 week after the initiation. CONCLUSIONS: These results successfully demonstrate the potential of the described in vivo and in vitro LSCM techniques to measure the angiogenesis process in pancreatic islet grafts.

Color as a source of information in the stereo correspondence process.

Although previous research has shown that depth perception is weak for isoluminant stereograms, the possibility remains that color plays an important role in stereopsis when luminance variations are present. To examine this possibility, we measured the relative contribution of chromatic and luminance cues in solving the correspondence problem for ambiguous "wallpaper" stereograms composed of vertical bars. Using an ideal-observer analysis, we found that chromatic cues were used much more efficiently than luminance cues in disambiguating these stereograms when the patterns were presented on a dark background but were used with about equal efficiency when presented on a light background. Another experiment (using the same wallpaper patterns) showed that chromatic and luminance cues were also used with about equal efficiency in a standard stereo detection task. Some of the implications of these results for theories of stereo vision are discussed.

The effect of median filtering on edge estimation and detection.

In this paper we consider the effect of median prefiltering on the subsequent estimation and detection of edges in digital images. Where possible, a quantitative statistical comparison is made for a number of filters defined with two-dimensional geometries; in some cases one-dimensional analyses are required to illustrate certain points. Noise images prefiltered by median filters defined with a variety of windowing geometries are used to support the analysis, and it is found that median prefiltering improves the performance of both thresholding and zero-crossing based edge detectors.

Embedded foveation image coding.

The human visual system (HVS) is highly space-variant in sampling, coding, processing, and understanding. The spatial resolution of the HVS is highest around the point of fixation (foveation point) and decreases rapidly with increasing eccentricity. By taking advantage of this fact, it is possible to remove considerable high-frequency information redundancy from the peripheral regions and still reconstruct a perceptually good quality image. Great success has been obtained previously by a class of embedded wavelet image coding algorithms, such as the embedded zerotree wavelet (EZW) and the set partitioning in hierarchical trees (SPIHT) algorithms. Embedded wavelet coding not only provides very good compression performance, but also has the property that the bitstream can be truncated at any point and still be decoded to recreate a reasonably good quality image. In this paper, we propose an embedded foveation image coding (EFIC) algorithm, which orders the encoded bitstream to optimize foveated visual quality at arbitrary bit-rates. A foveation-based image quality metric, namely, foveated wavelet image quality index (FWQI), plays an important role in the EFIC system. We also developed a modified SPIHT algorithm to improve the coding efficiency. Experiments show that EFIC integrates foveation filtering with foveated image coding and demonstrates very good coding performance and scalability in terms of foveated image quality measurement.

On eigenstructure-based direct multichannel blind image restoration.

Existing eigenstructure-based direct multichannel blind image restoration techniques include nullspace-based and direct deconvolver estimation techniques. The nullspace-based approach can be formulated as an optimization problem. We show that this formulation implies a new subspace-based approach that uses matrix operations. This new approach has the same advantages as the nullspace-based one but requires less computational complexity. Under some mild conditions, its complexity is equal to that of the FFT. Furthermore, the relation among the nullspace-based approach, the direct deconvolver estimation and the new subspace-based approach is studied.

Comments on "Subband coding of images using asymmetrical filterbanks".

In the above paper, Egger and Li presented a set of two-channel filterbanks, asymmetrical filterbanks (AFB's), for image coding applications. The basic properties of these filters are linear-phase, perfect reconstruction, asymmetric lengths for dual filters, and maximum regularity. In this correspondence, we point out that the proposed AFB's are not new in the sense that the proposed construction is equivalent to the factorization of Lagrange halfband filters, which has been reported by other researchers. In addition, we correct an error in the formulation of constructing AFBs in their paper.

Piecewise and local image models for regularized image restoration using cross-validation.

We describe two broad classes of useful and physically meaningful image models that can be used to construct novel smoothing constraints for use in the regularized image restoration problem. The two classes, termed piecewise image models (PIMs) and focal image models (LIMs), respectively, capture unique image properties that can be adapted to the image and that reflect structurally significant surface characteristics. Members of the PIM and LIM classes are easily formed into regularization operators that replace differential-type constraints. We also develop an adaptive strategy for selecting the best PIM or LIM for a given problem (from among the defined class), and we explain the construction of the corresponding regularization operators. Considerable attention is also given to determining the regularization parameter via a cross-validation technique, and also to the selection of an optimization strategy for solving the problem. Several results are provided that illustrate the processes of model selection, parameter selection, and image restoration. The overall approach provides a new viewpoint on the restoration problem through the use of new image models that capture salient image features that are not well represented through traditional approaches.

Nonlinear image estimation using piecewise and local image models.

We introduce a new approach to image estimation based on a flexible constraint framework that encapsulates meaningful structural image assumptions. Piecewise image models (PIMs) and local image models (LIMs) are defined and utilized to estimate noise-corrupted images, PIMs and LIMs are defined by image sets obeying certain piecewise or local image properties, such as piecewise linearity, or local monotonicity. By optimizing local image characteristics imposed by the models, image estimates are produced with respect to the characteristic sets defined by the models. Thus, we propose a new general formulation for nonlinear set-theoretic image estimation. Detailed image estimation algorithms and examples are given using two PIMs: piecewise constant (PICO) and piecewise linear (PILI) models, and two LIMs: locally monotonic (LOMO) and locally convex/concave (LOCO) models. These models define properties that hold over local image neighborhoods, and the corresponding image estimates may be inexpensively computed by iterative optimization algorithms. Forcing the model constraints to hold at every image coordinate of the solution defines a nonlinear regression problem that is generally nonconvex and combinatorial. However, approximate solutions may be computed in reasonable time using the novel generalized deterministic annealing (GDA) optimization technique, which is particularly well suited for locally constrained problems of this type. Results are given for corrupted imagery with signal-to-noise ratio (SNR) as low as 2 dB, demonstrating high quality image estimation as measured by local feature integrity, and improvement in SNR.

Multiresolution 3-D range segmentation using focus cues.

This paper describes a novel system for computing a three-dimensional (3-D) range segmentation of an arbitrary visible scene using focus information. The process of range segmentation is divided into three steps: an initial range classification, a surface merging process, and a 3-D multiresolution range segmentation. First, range classification is performed to obtain quantized range estimates. The range classification is performed by analyzing focus cues within a Bayesian estimation framework. A combined energy functional measures the degree of focus and the Gibbs distribution of the class field. The range classification provides an initial range segmentation. Second, a statistical merging process is performed to merge the initial surface segments. This gives a range segmentation at a coarse resolution. Third, 3-D multiresolution range segmentation (3-D MRS) is performed to refine the range segmentation into finer resolutions. The proposed range segmentation method does not require initial depth estimates, it allows the analysis of scenes containing multiple objects, and it provides a rich description of the 3-D structure of a scene.

Foveated video compression with optimal rate control.

Previously, fovcated video compression algorithms have been proposed which, in certain applications, deliver high-quality video at reduced bit rates by seeking to match the nonuniform sampling of the human retina. We describe such a framework here where foveated video is created by a nonuniform filtering scheme that increases the compressibility of the video stream. We maximize a new foveal visual quality metric. the foveal signal-to-noise ratio (FSNR) to determine the best compression and rate control parameters for a given target bit rate. Specifically, we establish a new optimal rate control algorithm for maximizing the FSNR using a Lagrange multiplier method defined on a curvilinear coordinate system. For optimal rate control, we also develop a piecewise R-D (rate-distortion)/R-Q (rate-quantization) model. A fast algorithm for searching for an optimal Lagrange multiplier lambda* is subsequently presented. For the new models, we show how the reconstructed video quality is affected, where the FSNR is maximized, and demonstrate the coding performance for H.263,+,++/MPEG-4 video coding. For H.263/MPEG video coding, a suboptimal rate control algorithm is developed for fast, high-performance applications. In the simulations, we compare the reconstructed pictures obtained using optimal rate control methods for foveated and normal video. We show that foveated video coding using the suboptimal rate control algorithm delivers excellent performance under 64 kb/s.

Stereoscopic ranging by matching image modulations.

We apply an AM-FM surface albedo model to analyze the projection of surface patterns viewed through a binocular camera system. This is used to support the use of modulation-based stereo matching where local image phase is used to compute stereo disparities. The local image phase is an advantageous feature for image matching, since the problem of computing disparities reduces to identifying local phase shifts between the stereoscopic image data. Local phase shifts, however, are problematic at high frequencies due to phase wrapping when disparities exceed +/-pi. We meld powerful multichannel Gabor image demodulation techniques for multiscale (coarse-to-fine) computation of local image phase with a disparity channel model for depth computation. The resulting framework unifies phase-based matching approaches with AM-FM surface/image models. We demonstrate the concepts in a stereo algorithm that generates a dense, accurate disparity map without the problems associated with phase wrapping.

Multidimensional quasi-eigenfunction approximations and multicomponent AM-FM models.

We develop multicomponent AM-FM models for multidimensional signals. The analysis is cast in a general n-dimensional framework where the component modulating functions are assumed to lie in certain Sobolev spaces. For both continuous and discrete linear shift invariant (LSI) systems with AM-FM inputs, powerful new approximations are introduced that provide closed form expressions for the responses in terms of the input modulations. The approximation errors are bounded by generalized energy variances quantifying the localization of the filter impulse response and by Sobolev norms quantifying the smoothness of the modulations. The approximations are then used to develop novel spatially localized demodulation algorithms that estimate the AM and FM functions for multiple signal components simultaneously from the channel responses of a multiband linear filterbank used to isolate components. Two discrete computational paradigms are presented. Dominant component analysis estimates the locally dominant modulations in a signal, which are useful in a variety of machine vision applications, while channelized components analysis delivers a true multidimensional multicomponent signal representation. We demonstrate the techniques on several images of general interest in practical applications, and obtain reconstructions that establish the validity of characterizing images of this type as sums of locally narrowband modulated components.

Modeling and quality assessment of halftoning by error diffusion.

Digital halftoning quantizes a graylevel image to one bit per pixel. Halftoning by error diffusion reduces local quantization error by filtering the quantization error in a feedback loop. In this paper, we linearize error diffusion algorithms by modeling the quantizer as a linear gain plus additive noise. We confirm the accuracy of the linear model in three independent ways. Using the linear model, we quantify the two primary effects of error diffusion: edge sharpening and noise shaping. For each effect, we develop an objective measure of its impact on the subjective quality of the halftone. Edge sharpening is proportional to the linear gain, and we give a formula to estimate the gain from a given error filter. In quantifying the noise, we modify the input image to compensate for the sharpening distortion and apply a perceptually weighted signal-to-noise ratio to the residual of the halftone and modified input image. We compute the correlation between the residual and the original image to show when the residual can be considered signal independent. We also compute a tonality measure similar to total harmonic distortion. We use the proposed measures for edge sharpening, noise shaping, and tonality to evaluate the quality of error diffusion algorithms.

The multicomponent AM-FM image representation.

We compute AM-FM representations for multicomponent, nonstationary images using a statistical component model. Components are isolated with a filterbank comprising frequency and orientation selective channels. The modulating functions for each component are estimated from the channel responses using localized nonlinear operators followed by optimal MMSE estimators. We also demonstrate reconstruction from the representation.

Multidimensional orthogonal FM transforms.

The present a novel class of multidimensional orthogonal FM transforms. The analysis suggests a novel signal-adaptive FM transform possessing interesting energy compaction properties. We show that the proposed signal-adaptive FM transform produces point spectra for multidimensional signals with uniformly distributed samples. This suggests that the proposed transform is suitable for energy compaction and subsequent coding of broadband signals and images that locally exhibit significant level diversity. We illustrate these concepts with simulation experiments.

Image quality assessment based on a degradation model.

We model a degraded image as an original image that has been subject to linear frequency distortion and additive noise injection. Since the psychovisual effects of frequency distortion and noise injection are independent, we decouple these two sources of degradation and measure their effect on the human visual system. We develop a distortion measure (DM) of the effect of frequency distortion, and a noise quality measure (NQM) of the effect of additive noise. The NQM, which is based on Peli's (1990) contrast pyramid, takes into account the following: 1) variation in contrast sensitivity with distance, image dimensions, and spatial frequency; 2) variation in the local luminance mean; 3) contrast interaction between spatial frequencies; 4) contrast masking effects. For additive noise, we demonstrate that the nonlinear NQM is a better measure of visual quality than peak signal-to noise ratio (PSNR) and linear quality measures. We compute the DM in three steps. First, we find the frequency distortion in the degraded image. Second, we compute the deviation of this frequency distortion from an allpass response of unity gain (no distortion). Finally, we weight the deviation by a model of the frequency response of the human visual system and integrate over the visible frequencies. We demonstrate how to decouple distortion and additive noise degradation in a practical image restoration system.

A fast, high-quality inverse halftoning algorithm for error diffused halftones.

Halftones and other binary images are difficult to process with causing several degradation. Degradation is greatly reduced if the halftone is inverse halftoned (converted to grayscale) before scaling, sharpening, rotating, or other processing. For error diffused halftones, we present (1) a fast inverse halftoning algorithm and (2) a new multiscale gradient estimator. The inverse halftoning algorithm is based on anisotropic diffusion. It uses the new multiscale gradient estimator to vary the tradeoff between spatial resolution and grayscale resolution at each pixel to obtain a sharp image with a low perceived noise level. Because the algorithm requires fewer than 300 arithmetic operations per pixel and processes 7x7 neighborhoods of halftone pixels, it is well suited for implementation in VLSI and embedded software. We compare the implementation cost, peak signal to noise ratio, and visual quality with other inverse halftoning algorithms.

Generalized deterministic annealing.

We develop a general formalism for computing high quality, low-cost solutions to nonconvex combinatorial optimization problems expressible as distributed interacting local constraints. For problems of this type, generalized deterministic annealing (GDA) avoids the performance-related sacrifices of current techniques. GDA exploits the localized structure of such problems by assigning K-state neurons to each optimization variable. The neuron values correspond to the probability densities of K-state local Markov chains and may be updated serially or in parallel; the Markov model is derived from the Markov model of simulated annealing (SA), although it is greatly simplified. Theorems are presented that firmly establish the convergence properties of GDA, as well as supplying practical guidelines for selecting the initial and final temperatures in the annealing process. A benchmark image enhancement application is provided where the performance of GDA is compared to other optimization methods. The empirical data taken in conjunction with the formal analytical results suggest that GDA enjoys significant performance advantages relative to current methods for combinatorial optimization.