Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation.

Optical flow has made great progress in clean scenes, while suffers degradation under adverse weather due to the violation of the brightness constancy and gradient continuity assumptions of optical flow. Typically, existing methods mainly adopt domain adaptation to transfer motion knowledge from clean to degraded domain through one-stage adaptation. However, this direct adaptation is ineffective, since there exists a large gap due to adverse weather and scene style between clean and real degraded domains. Moreover, even within the degraded domain itself, static weather (e.g., fog) and dynamic weather (e.g., rain) have different impacts on optical flow. To address above issues, we explore synthetic degraded domain as an intermediate bridge between clean and real degraded domains, and propose a cumulative homogeneous-heterogeneous adaptation framework for real adverse weather optical flow. Specifically, for clean-degraded transfer, our key insight is that static weather possesses the depth-association homogeneous feature which does not change the intrinsic motion of the scene, while dynamic weather additionally introduces the heterogeneous feature which results in a significant boundary discrepancy in warp errors between clean and degraded domains. For synthetic-real transfer, we figure out that cost volume correlation shares a similar statistical histogram between synthetic and real degraded domains, benefiting to holistically aligning the homogeneous correlation distribution for synthetic-real knowledge distillation. Under this unified framework, the proposed method can progressively and explicitly transfer knowledge from clean scenes to real adverse weather. In addition, we further collect a real adverse weather dataset with manually annotated optical flow labels and perform extensive experiments to verify the superiority of the proposed method. Both the code and the dataset will be available at https://github.com/hyzhouboy/CH2DA-Flow.

I2C: Invertible Continuous Codec for High-Fidelity Variable-Rate Image Compression.

Lossy image compression is a fundamental technology in media transmission and storage. Variable-rate approaches have recently gained much attention to avoid the usage of a set of different models for compressing images at different rates. During the media sharing, multiple re-encodings with different rates would be inevitably executed. However, existing Variational Autoencoder (VAE)-based approaches would be readily corrupted in such circumstances, resulting in the occurrence of strong artifacts and the destruction of image fidelity. Based on the theoretical findings of preserving image fidelity via invertible transformation, we aim to tackle the issue of high-fidelity fine variable-rate image compression and thus propose the Invertible Continuous Codec (I2C). We implement the I2C in a mathematical invertible manner with the core Invertible Activation Transformation (IAT) module. I2C is constructed upon a single-rate Invertible Neural Network (INN) based model and the quality level (QLevel) would be fed into the IAT to generate scaling and bias tensors. Extensive experiments demonstrate that the proposed I2C method outperforms state-of-the-art variable-rate image compression methods by a large margin, especially after multiple continuous re-encodings with different rates, while having the ability to obtain a very fine variable-rate control without any performance compromise.

Unsupervised Deraining: Where Asymmetric Contrastive Learning Meets Self-Similarity.

Most existing learning-based deraining methods are supervisedly trained on synthetic rainy-clean pairs. The domain gap between the synthetic and real rain makes them less generalized to complex real rainy scenes. Moreover, the existing methods mainly utilize the property of the image or rain layers independently, while few of them have considered their mutually exclusive relationship. To solve above dilemma, we explore the intrinsic intra-similarity within each layer and inter-exclusiveness between two layers and propose an unsupervised non-local contrastive learning (NLCL) deraining method. The non-local self-similarity image patches as the positives are tightly pulled together and rain patches as the negatives are remarkably pushed away, and vice versa. On one hand, the intrinsic self-similarity knowledge within positive/negative samples of each layer benefits us to discover more compact representation; on the other hand, the mutually exclusive property between the two layers enriches the discriminative decomposition. Thus, the internal self-similarity within each layer (similarity) and the external exclusive relationship of the two layers (dissimilarity) serving as a generic image prior jointly facilitate us to unsupervisedly differentiate the rain from clean image. We further discover that the intrinsic dimension of the non-local image patches is generally higher than that of the rain patches. This insight motivates us to design an asymmetric contrastive loss that precisely models the compactness discrepancy of the two layers, thereby improving the discriminative decomposition. In addition, recognizing the limited quality of existing real rain datasets, which are often small-scale or obtained from the internet, we collect a large-scale real dataset under various rainy weathers that contains high-resolution rainy images. Extensive experiments conducted on different real rainy datasets demonstrate that the proposed method obtains state-of-the-art performance in real deraining.

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy.

Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

Nonlinear Deblurring for Low-Light Saturated Image.

Single image deblurring has achieved significant progress for natural daytime images. Saturation is a common phenomenon in blurry images, due to the low light conditions and long exposure times. However, conventional linear deblurring methods usually deal with natural blurry images well but result in severe ringing artifacts when recovering low-light saturated blurry images. To solve this problem, we formulate the saturation deblurring problem as a nonlinear model, in which all the saturated and unsaturated pixels are modeled adaptively. Specifically, we additionally introduce a nonlinear function to the convolution operator to accommodate the procedure of the saturation in the presence of the blurring. The proposed method has two advantages over previous methods. On the one hand, the proposed method achieves the same high quality of restoring the natural image as seen in conventional deblurring methods, while also reducing the estimation errors in saturated areas and suppressing ringing artifacts. On the other hand, compared with the recent saturated-based deblurring methods, the proposed method captures the formation of unsaturated and saturated degradations straightforwardly rather than with cumbersome and error-prone detection steps. Note that, this nonlinear degradation model can be naturally formulated into a maximum-a posterioriframework, and can be efficiently decoupled into several solvable sub-problems via the alternating direction method of multipliers (ADMM). Experimental results on both synthetic and real-world images demonstrate that the proposed deblurring algorithm outperforms the state-of-the-art low-light saturation-based deblurring methods.

Direction and Residual Awareness Curriculum Learning Network for Rain Streaks Removal.

Single-image rain streaks' removal has attracted great attention in recent years. However, due to the highly visual similarity between the rain streaks and the line pattern image edges, the over-smoothing of image edges or residual rain streaks' phenomenon may unexpectedly occur in the deraining results. To overcome this problem, we propose a direction and residual awareness network within the curriculum learning paradigm for the rain streaks' removal. Specifically, we present a statistical analysis of the rain streaks on large-scale real rainy images and figure out that rain streaks in local patches possess principal directionality. This motivates us to design a direction-aware network for rain streaks' modeling, in which the principal directionality property endows us with the discriminative representation ability of better differing rain streaks from image edges. On the other hand, for image modeling, we are motivated by the iterative regularization in classical image processing and unfold it into a novel residual-aware block (RAB) to explicitly model the relationship between the image and the residual. The RAB adaptively learns balance parameters to selectively emphasize informative image features and better suppress the rain streaks. Finally, we formulate the rain streaks' removal problem into the curriculum learning paradigm which progressively learns the directionality of the rain streaks, rain streaks' appearance, and the image layer in a coarse-to-fine, easy-to-hard guidance manner. Solid experiments on extensive simulated and real benchmarks demonstrate the visual and quantitative improvement of the proposed method over the state-of-the-art methods.

Learning Oriented Object Detection via Naive Geometric Computing.

Detecting oriented objects along with estimating their rotation information is one crucial step for image analysis, especially for remote sensing images. Despite that many methods proposed recently have achieved remarkable performance, most of them directly learn to predict object directions under the supervision of only one (e.g., the rotation angle) or a few (e.g., several coordinates) groundtruth (GT) values individually. Oriented object detection would be more accurate and robust if extra constraints, with respect to proposal and rotation information regression, are adopted for joint supervision during training. To this end, we propose a mechanism that simultaneously learns the regression of horizontal proposals, oriented proposals, and rotation angles of objects in a consistent manner, via naive geometric computing, as one additional steady constraint. An oriented center prior guided label assignment strategy is proposed for further enhancing the quality of proposals, yielding better performance. Extensive experiments on six datasets demonstrate the model equipped with our idea significantly outperforms the baseline by a large margin and several new state-of-the-art results are achieved without any extra computational burden during inference. Our proposed idea is simple and intuitive that can be readily implemented. Source codes are publicly available at: https://github.com/wangWilson/CGCDet.git.

Improvement of Colloidal Characteristics in a Precursor Solution by a PbI2-(DMSO)2 Complex for Efficient Nonstoichiometrically Prepared CsPbI2.8Br0.2 Perovskite Solar Cells.

The optoelectronic properties of all-inorganic perovskite solar cells are greatly affected by the quality characteristics of films, such as the defect concentration, crystal growth orientation, crystallinity, and morphology. In this study, a PbI2-(DMSO)2 complex is adopted to partially replace PbI2 as the lead source in the preparation of perovskite precursor solutions. Due to the rapid dispersion of the PbI2-(DMSO)2 complex in a solvent, raw materials can rapidly react to form perovskite colloids with a narrow size distribution. Such uniform colloidal particles are found to be beneficial for achieving films with improved quality and highly orientated growth along the [001] direction. The optimized film exhibits a clearly improved crystallinity and a decrease in defect concentration from 4.29 × 1015 cm-3 to 3.20 × 1015 cm-3. The device based on the obtained all-inorganic CsPbI2.8Br0.2 perovskite finally achieves an increase in photovoltaic power conversion efficiency from 10.5 to 14.15%. In addition, the environmental stability of the device also benefits from the improved film quality. After 480 h of storage in air, the device can still maintain nearly 80% of its initial performance.

Weighted Low-Rank Tensor Recovery for Hyperspectral Image Restoration.

Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such as noises (random noise), blurs (Gaussian and uniform blur), and downsampled (both spectral and spatial downsample), each corresponding to the HSI denoising, deblurring, and super-resolution tasks, respectively. Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit the structurally spectral-spatial correlation in 3-D HSI. To overcome these limitations, in this article, we propose a unified low-rank tensor recovery model for comprehensive HSI restoration tasks, in which nonlocal similarity within spectral-spatial cubic and spectral correlation are simultaneously captured by third-order tensors. Furthermore, to improve the capability and flexibility, we formulate it as a weighted low-rank tensor recovery (WLRTR) model by treating the singular values differently. We demonstrate the reweighed strategy, which has been extensively studied in the matrix, also greatly benefits the tensor modeling. We also consider the stripe noise in HSI as the sparse error by extending WLRTR to robust principal component analysis (WLRTR-RPCA). Extensive experiments demonstrate the proposed WLRTR models consistently outperform state-of-the-art methods in typical HSI low-level vision tasks, including denoising, destriping, deblurring, and super-resolution.

A variation-based ring artifact correction method with sparse constraint for flat-detector CT.

The reconstructed slice quality of flat-detector computed tomography (CT) is often disturbed by concentric-ring artifacts. Since concentric rings in CT slices appear as straight stripes when transformed into polar coordinates, a variation-based model is proposed to suppress the stripes. The method is motivated by two observations about stripes in polar coordinates: (1) ring artifacts attenuate gradually along the radial direction, leading to a sparse distribution of stripes and (2) stripes greatly distort the image gradient across the stripes, while slightly affecting the image gradient along the stripes. Thus, a [Formula: see text]-norm-based data fidelity term and a [Formula: see text]-norm/[Formula: see text]-norm unidirectional variation-based regularization term are presented to characterize the stripes. The alternating direction method of multipliers is introduced to solve the resulting minimization problem. Moreover, we discuss the interpolation methods used in coordinate transformation and find that the nearest neighbor interpolation is optimal. Experimental results on simulated and real data demonstrate that our method can correct ring artifacts effectively compared with state-of-the-art coordinate transformation-based methods, as well as preserve the structures and details of slices.

Joint baseline-correction and denoising for Raman spectra.

Laser instruments often suffer from the problem of baseline drift and random noise, which greatly degrade spectral quality. In this article, we propose a variation model that combines baseline correction and denoising. First, to guide the baseline estimation, morphological operations are adopted to extract the characteristics of the degraded spectrum. Second, to suppress noise in both the spectrum and baseline, Tikhonov regularization is introduced. Moreover, we describe an efficient optimization scheme that alternates between the latent spectrum estimation and the baseline correction until convergence. The major novel aspect of the proposed algorithms is the estimation of a smooth spectrum and removal of the baseline simultaneously. Results of a comparison with state-of-the-art methods demonstrate that the proposed method outperforms them in both qualitative and quantitative assessments.

Anisotropic spectral-spatial total variation model for multispectral remote sensing image destriping.

Multispectral remote sensing images often suffer from the common problem of stripe noise, which greatly degrades the imaging quality and limits the precision of the subsequent processing. The conventional destriping approaches usually remove stripe noise band by band, and show their limitations on different types of stripe noise. In this paper, we tentatively categorize the stripes in remote sensing images in a more comprehensive manner. We propose to treat the multispectral images as a spectral-spatial volume and pose an anisotropic spectral-spatial total variation regularization to enhance the smoothness of solution along both the spectral and spatial dimension. As a result, a more comprehensive stripes and random noise are perfectly removed, while the edges and detail information are well preserved. In addition, the split Bregman iteration method is employed to solve the resulting minimization problem, which highly reduces the computational load. We extensively validate our method under various stripe categories and show comparison with other approaches with respect to result quality, running time, and quantitative assessments.

Robust destriping method with unidirectional total variation and framelet regularization.

Multidetector imaging systems often suffer from the problem of stripe noise and random noise, which greatly degrade the imaging quality. In this paper, we propose a variational destriping method that combines unidirectional total variation and framelet regularization. Total-variation-based regularizations are considered effective in removing different kinds of stripe noise, and framelet regularization can efficiently preserve the detail information. In essence, these two regularizations are complementary to each other. Moreover, the proposed method can also efficiently suppress random noise. The split Bregman iteration method is employed to solve the resulting minimization problem. Comparative results demonstrate that the proposed method significantly outperforms state-of-the-art destriping methods on both qualitative and quantitative assessments.

Blind Poissonian images deconvolution with framelet regularization.

We propose a maximum a posteriori blind Poissonian images deconvolution approach with framelet regularization for the image and total variation (TV) regularization for the point spread function. Compared with the TV based methods, our algorithm not only suppresses noise effectively but also recovers edges and detailed information. Moreover, the split Bregman method is exploited to solve the resulting minimization problem. Comparative results on both simulated and real images are reported.

Spatial-spectral method for classification of hyperspectral images.

Spatial-spectral approach with spatially adaptive classification of hyperspectral images is proposed. The rotation-invariant spatial texture information for each object is exploited and incorporated into the classifier by using the modified local Gabor binary pattern to distinguish different types of classes of interest. The proposed method can effectively suppress anisotropic texture in spatially separate classes as well as improve the discrimination among classes. Moreover, it becomes more robust with the within-class variation. Experimental results on the classification of three real hyperspectral remote sensing images demonstrate the effectiveness of the proposed approach.

Semi-blind spectral deconvolution with adaptive Tikhonov regularization.

Deconvolution has become one of the most used methods for improving spectral resolution. Deconvolution is an ill-posed problem, especially when the point spread function (PSF) is unknown. Non-blind deconvolution methods use a predefined PSF, but in practice the PSF is not known exactly. Blind deconvolution methods estimate the PSF and spectrum simultaneously from the observed spectra, which become even more difficult in the presence of strong noise. In this paper, we present a semi-blind deconvolution method to improve the spectral resolution that does not assume a known PSF but models it as a parametric function in combination with the a priori knowledge about the characteristics of the instrumental response. First, we construct the energy functional, including Tikhonov regularization terms for both the spectrum and the parametric PSF. Moreover, an adaptive weighting term is devised in terms of the magnitude of the first derivative of spectral data to adjust the Tikhonov regularization for the spectrum. Then we minimize the energy functional to obtain the spectrum and the parameters of the PSF. We also discuss how to select the regularization parameters. Comparative results with other deconvolution methods on simulated degraded spectra, as well as on experimental infrared spectra, are presented.

A MAP-based algorithm for spectroscopic semi-blind deconvolution.

Spectroscopic data often suffer from common problems of bands overlapping and random noise. In this paper, we show that the issue of overlapping peaks can be considered as a maximum a posterior (MAP) problem and be solved by minimizing an object functional that includes a likelihood term and two prior terms. In the MAP framework, the likelihood probability density function (PDF) is constructed based on a spectral observation model, a robust Huber-Markov model is used as spectra prior PDF, and the kernel prior is described based on a parametric Gaussian function. Moreover, we describe an efficient optimization scheme that alternates between latent spectrum recovery and blur kernel estimation until convergence. The major novelty of the proposed algorithm is that it can estimate the kernel slit width and latent spectrum simultaneously. Comparative results with other deconvolution methods suggest that the proposed method can recover spectral structural details as well as suppress noise effectively.

Blind image deconvolution with spatially adaptive total variation regularization.

A blind deconvolution algorithm with spatially adaptive total variation regularization is introduced. The spatial information in different image regions is incorporated into regularization by using the edge indicator called difference eigenvalue to distinguish edges from flat areas. The proposed algorithm can effectively reduce the noise in flat regions as well as preserve the edge and detailed information. Moreover, it becomes more robust with the change of the regularization parameter. Comparative results on simulated and real degraded images are reported.

Saliency model for object detection: searching for novel items in the scene.

This Letter presents a new computational model of visual saliency. A new definition for saliency is proposed: saliency is novelty, which guides the deployment of visual attention. We define novelty as coming from regions that contain dissimilarities from the global scene. Our approach consists of two stages: First, obtain a global perspective. The global representation is obtained with a visual vocabulary. A novelty factor for each visual word is introduced according to the "repetition suppression principle." Second, obtain a local perspective. A local representation is achieved from the histogram of visual word occurrence. The metric of saliency is defined as the overall novelty factor of the visual words. Experimental results demonstrate good performance of the proposed model on complex scenes and fair consistency with human eye fixation data.

X-ray angiogram images enhancement by facet-based adaptive anisotropic diffusion.

The paper presents a versatile nonlinear diffusion method to visually enhance the angiogram images for improving the clinical diagnosis. Traditional nonlinear diffusion has been shown very effective in edge-preserved smoothing of images. However, the existing nonlinear diffusion models suffer several drawbacks: sensitivity to the choice of the conductance parameter, limited range of edge enhancement, and the sensitivity to the selection of evolution time. The new anisotropic diffusion we proposed is based on facet model which can solve the issues mentioned above adaptively according to the image content. This method uses facet model for fitting the image to reduce noise, and uses the sum square of eigenvalues of Hessian as the standard of the conductance parameter selection synchronously. The capability of dealing with noise and conductance parameter can also change adaptively in the whole diffusion process. Moreover, our method is not sensitive to the choice of evolution time. Experimental results show that our new method is more effective than the original anisotropic diffusion.