Low-Rank Tensor Function Representation for Multi-Dimensional Data Recovery.

Since higher-order tensors are naturally suitable for representing multi-dimensional data in real-world, e.g., color images and videos, low-rank tensor representation has become one of the emerging areas in machine learning and computer vision. However, classical low-rank tensor representations can solely represent multi-dimensional discrete data on meshgrid, which hinders their potential applicability in many scenarios beyond meshgrid. To break this barrier, we propose a low-rank tensor function representation (LRTFR) parameterized by multilayer perceptrons (MLPs), which can continuously represent data beyond meshgrid with powerful representation abilities. Specifically, the suggested tensor function, which maps an arbitrary coordinate to the corresponding value, can continuously represent data in an infinite real space. Parallel to discrete tensors, we develop two fundamental concepts for tensor functions, i.e., the tensor function rank and low-rank tensor function factorization, and utilize MLPs to paramterize factor functions of the tensor function factorization. We theoretically justify that both low-rank and smooth regularizations are harmoniously unified in LRTFR, which leads to high effectiveness and efficiency for data continuous representation. Extensive multi-dimensional data recovery applications arising from image processing (image inpainting and denoising), machine learning (hyperparameter optimization), and computer graphics (point cloud upsampling) substantiate the superiority and versatility of our method as compared with state-of-the-art methods. Especially, the experiments beyond the original meshgrid resolution (hyperparameter optimization) or even beyond meshgrid (point cloud upsampling) validate the favorable performances of our method for continuous representation.

Eigenimage2Eigenimage (E2E): A Self-Supervised Deep Learning Network for Hyperspectral Image Denoising.

The performance of deep learning-based denoisers highly depends on the quantity and quality of training data. However, paired noisy-clean training images are generally unavailable in hyperspectral remote sensing areas. To solve this problem, this work resorts to the self-supervised learning technique, where our proposed model can train itself to learn one part of noisy input from another part of noisy input. We study a general hyperspectral image (HSI) denoising framework, called Eigenimage2Eigenimage (E2E), which turns the HSI denoising problem into an eigenimage (i.e., the subspace representation coefficients of the HSI) denoising problem and proposes a learning strategy to generate noisy-noisy paired training eigenimages from noisy eigenimages. Consequently, the E2E denoising framework can be trained without clean data and applied to denoise HSIs without the constraint with the number of frequency bands. Experimental results are provided to demonstrate the performance of the proposed method that is better than the other existing deep learning methods for denoising HSIs. A MATLAB demo of this work is available at https://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimagehttps://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimage for the sake of reproducibility.

Transfer Learning With Singular Value Decomposition of Multichannel Convolution Matrices.

The task of transfer learning using pretrained convolutional neural networks is considered. We propose a convolution-SVD layer to analyze the convolution operators with a singular value decomposition computed in the Fourier domain. Singular vectors extracted from the source domain are transferred to the target domain, whereas the singular values are fine-tuned with a target data set. In this way, dimension reduction is achieved to avoid overfitting, while some flexibility to fine-tune the convolution kernels is maintained. We extend an existing convolution kernel reconstruction algorithm to allow for a reconstruction from an arbitrary set of learned singular values. A generalization bound for a single convolution-SVD layer is devised to show the consistency between training and testing errors. We further introduce a notion of transfer learning gap. We prove that the testing error for a single convolution-SVD layer is bounded in terms of the gap, which motivates us to develop a regularization model with the gap as the regularizer. Numerical experiments are conducted to demonstrate the superiority of the proposed model in solving classification problems and the influence of various parameters. In particular, the regularization is shown to yield a significantly higher prediction accuracy.

Multi-relational graph convolutional networks: Generalization guarantees and experiments.

The class of multi-relational graph convolutional networks (MRGCNs) is a recent extension of standard graph convolutional networks (GCNs) to handle heterogenous graphs with multiple types of relationships. MRGCNs have been shown to yield results superior than traditional GCNs in various machine learning tasks. The key idea is to introduce a new kind of convolution operated on tensors that can effectively exploit correlations exhibited in multiple relationships. The main objective of this paper is to analyze the algorithmic stability and generalization guarantees of MRGCNs to confirm the usefulness of MRGCNs. Our contributions are of three folds. First, we develop a matrix representation of various tensor operations underneath MRGCNs to simplify the analysis significantly. Next, we prove the uniform stability of MRGCNs and deduce the convergence of the generalization gap to support the usefulness of MRGCNs. The analysis sheds lights on the design of MRGCNs, for instance, how the data should be scaled to achieve the uniform stability of the learning process. Finally, we provide experimental results to demonstrate the stability results.

Tucker network: Expressive power and comparison.

Deep neural networks have achieved great success in solving many machine learning and computer vision problems. In this paper, we propose a deep neural network called the Tucker network derived from the Tucker format and analyze its expressive power. The results demonstrate that the Tucker network has exponentially higher expressive power than the shallow network. In other words, a shallow network with an exponential width is required to realize the same score function as that computed by the Tucker network. Moreover, we discuss the expressive power between the hierarchical Tucker tensor network (HT network) and the proposed Tucker network. To generalize the Tucker network into a deep version, we combine the hierarchical Tucker format and Tucker format to propose a deep Tucker tensor decomposition. Its corresponding deep Tucker network is presented. Experiments are conducted on three datasets: MNIST, CIFAR-10 and CIFAR-100. The results experimentally validate the theoretical results and show that the Tucker network and deep Tucker network have better performance than the shallow network and HT network.

FastHyMix: Fast and Parameter-Free Hyperspectral Image Mixed Noise Removal.

The decrease in the widths of spectral bands in hyperspectral imaging leads to a decrease in signal-to-noise ratio (SNR) of measurements. The decreased SNR reduces the reliability of measured features or information extracted from hyperspectral images (HSIs). Furthermore, the image degradations linked with various mechanisms also result in different types of noise, such as Gaussian noise, impulse noise, deadlines, and stripes. This article introduces a fast and parameter-free hyperspectral image mixed noise removal method (termed FastHyMix), which characterizes the complex distribution of mixed noise by using a Gaussian mixture model and exploits two main characteristics of hyperspectral data, namely, low rankness in the spectral domain and high correlation in the spatial domain. The Gaussian mixture model enables us to make a good estimation of Gaussian noise intensity and the locations of sparse noise. The proposed method takes advantage of the low rankness using subspace representation and the spatial correlation of HSIs by adding a powerful deep image prior, which is extracted from a neural denoising network. An exhaustive array of experiments and comparisons with state-of-the-art denoisers was carried out. The experimental results show significant improvement in both synthetic and real datasets. A MATLAB demo of this work is available at https://github.com/LinaZhuang for the sake of reproducibility.

Dictionary Learning With Low-Rank Coding Coefficients for Tensor Completion.

In this article, we propose a novel tensor learning and coding model for third-order data completion. The aim of our model is to learn a data-adaptive dictionary from given observations and determine the coding coefficients of third-order tensor tubes. In the completion process, we minimize the low-rankness of each tensor slice containing the coding coefficients. By comparison with the traditional predefined transform basis, the advantages of the proposed model are that: 1) the dictionary can be learned based on the given data observations so that the basis can be more adaptively and accurately constructed and 2) the low-rankness of the coding coefficients can allow the linear combination of dictionary features more effectively. Also we develop a multiblock proximal alternating minimization algorithm for solving such tensor learning and coding model and show that the sequence generated by the algorithm can globally converge to a critical point. Extensive experimental results for real datasets such as videos, hyperspectral images, and traffic data are reported to demonstrate these advantages and show that the performance of the proposed tensor learning and coding method is significantly better than the other tensor completion methods in terms of several evaluation metrics.

Morphological Feature Visualization of Alzheimer's Disease via Multidirectional Perception GAN.

The diagnosis of early stages of Alzheimer's disease (AD) is essential for timely treatment to slow further deterioration. Visualizing the morphological features for early stages of AD is of great clinical value. In this work, a novel multidirectional perception generative adversarial network (MP-GAN) is proposed to visualize the morphological features indicating the severity of AD for patients of different stages. Specifically, by introducing a novel multidirectional mapping mechanism into the model, the proposed MP-GAN can capture the salient global features efficiently. Thus, using the class discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the predefined target domain. Besides, by integrating the adversarial loss, classification loss, cycle consistency loss, and L1 penalty, a single generator in MP-GAN can learn the class discriminative maps for multiple classes. Extensive experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate that MP-GAN achieves superior performance compared with the existing methods. The lesions visualized by MP-GAN are also consistent with what clinicians observe.

Self-Supervised Nonlinear Transform-Based Tensor Nuclear Norm for Multi-Dimensional Image Recovery.

Recently, transform-based tensor nuclear norm (TNN) minimization methods have received increasing attention for recovering third-order tensors in multi-dimensional imaging problems. The main idea of these methods is to perform the linear transform along the third mode of third-order tensors and then minimize the nuclear norm of frontal slices of the transformed tensor. The main aim of this paper is to propose a nonlinear multilayer neural network to learn a nonlinear transform by solely using the observed tensor in a self-supervised manner. The proposed network makes use of the low-rank representation of the transformed tensor and data-fitting between the observed tensor and the reconstructed tensor to learn the nonlinear transform. Extensive experimental results on different data and different tasks including tensor completion, background subtraction, robust tensor completion, and snapshot compressive imaging demonstrate the superior performance of the proposed method over state-of-the-art methods.

Non-Local Robust Quaternion Matrix Completion for Large-Scale Color Image and Video Inpainting.

The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image and has been widely applied in many recently proposed machining learning algorithms for image processing. However, there is no theoretical analysis on its working principle in the literature. In this paper, we discover a potential causality between NSS and low-rank property of color images, which is also available to grey images. A new patch group based NSS prior scheme is proposed to learn explicit NSS models of natural color images. The numerical low-rank property of patched matrices is also rigorously proved. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new tensor NSS-based QMC method is also presented to solve the color video inpainting problem based on quaternion tensor representation. The numerical experiments on color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods.

Tensor Completion via Complementary Global, Local, and Nonlocal Priors.

Completing missing entries in multidimensional visual data is a typical ill-posed problem that requires appropriate exploitation of prior information of the underlying data. Commonly used priors can be roughly categorized into three classes: global tensor low-rankness, local properties, and nonlocal self-similarity (NSS); most existing works utilize one or two of them to implement completion. Naturally, there arises an interesting question: can one concurrently make use of multiple priors in a unified way, such that they can collaborate with each other to achieve better performance? This work gives a positive answer by formulating a novel tensor completion framework which can simultaneously take advantage of the global-local-nonlocal priors. In the proposed framework, the tensor train (TT) rank is adopted to characterize the global correlation; meanwhile, two Plug-and-Play (PnP) denoisers, including a convolutional neural network (CNN) denoiser and the color block-matching and 3 D filtering (CBM3D) denoiser, are incorporated to preserve local details and exploit NSS, respectively. Then, we design a proximal alternating minimization algorithm to efficiently solve this model under the PnP framework. Under mild conditions, we establish the convergence guarantee of the proposed algorithm. Extensive experiments show that these priors organically benefit from each other to achieve state-of-the-art performance both quantitatively and qualitatively.

Low Rank Tensor Completion With Poisson Observations.

Poisson observations for videos are important models in video processing and computer vision. In this paper, we study the third-order tensor completion problem with Poisson observations. The main aim is to recover a tensor based on a small number of its Poisson observation entries. A existing matrix-based method may be applied to this problem via the matricized version of the tensor. However, this method does not leverage on the global low-rankness of a tensor and may be substantially suboptimal. Our approach is to consider the maximum likelihood estimate of the Poisson distribution, and utilize the Kullback-Leibler divergence for the data-fitting term to measure the observations and the underlying tensor. Moreover, we propose to employ a transformed tensor nuclear norm ball constraint and a bounded constraint of each entry, where the transformed tensor nuclear norm is used to get a lower transformed multi-rank tensor with suitable unitary transformation matrices. We show that the upper bound of the error of the estimator of the proposed model is less than that of the existing matrix-based method. Also an information theoretic lower error bound is established. An alternating direction method of multipliers is developed to solve the resulting convex optimization model. Extensive numerical experiments on synthetic data and real-world datasets are presented to demonstrate the effectiveness of our proposed model compared with existing tensor completion methods.

Tensorizing GAN With High-Order Pooling for Alzheimer's Disease Assessment.

It is of great significance to apply deep learning for the early diagnosis of Alzheimer's disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess mild cognitive impairment (MCI) and AD. By tensorizing a three-player cooperative game-based framework, the proposed model can benefit from the structural information of the brain. By incorporating the high-order pooling scheme into the classifier, the proposed model can make full use of the second-order statistics of holistic magnetic resonance imaging (MRI). To the best of our knowledge, the proposed Tensor-train, High-order pooling and Semisupervised learning-based GAN (THS-GAN) is the first work to deal with classification on MR images for AD diagnosis. Extensive experimental results on Alzheimer's disease neuroimaging initiative (ADNI) data set are reported to demonstrate that the proposed THS-GAN achieves superior performance compared with existing methods, and to show that both tensor-train and high-order pooling can enhance classification performance. The visualization of generated samples also shows that the proposed model can generate plausible samples for semisupervised learning purpose.

Multiscale Feature Tensor Train Rank Minimization for Multidimensional Image Recovery.

The general tensor-based methods can recover missing values of multidimensional images by exploiting the low-rankness on the pixel level. However, especially when considerable pixels of an image are missing, the low-rankness is not reliable on the pixel level, resulting in some details losing in their results, which hinders the performance of subsequent image applications (e.g., image recognition and segmentation). In this article, we suggest a novel multiscale feature (MSF) tensorization by exploiting the MSFs of multidimensional images, which not only helps to recover the missing values on a higher level, that is, the feature level but also benefits subsequent image applications. By exploiting the low-rankness of the resulting MSF tensor constructed by the new tensorization, we propose the convex and nonconvex MSF tensor train rank minimization (MSF-TT) to conjointly recover the MSF tensor and the corresponding original tensor in a unified framework. We develop the alternating directional method of multipliers (ADMMs) to solve the convex MSF-TT and the proximal alternating minimization (PAM) to solve the nonconvex MSF-TT. Moreover, we establish the theoretical guarantee of convergence for the PAM algorithm. Numerical examples of real-world multidimensional images show that the proposed MSF-TT outperforms other compared approaches in image recovery and the recovered MSF tensor can benefit the subsequent image recognition.

Heterogeneous Domain Adaptation by Information Capturing and Distribution Matching.

Heterogeneous domain adaptation (HDA) is a challenging problem because of the different feature representations in the source and target domains. Most HDA methods search for mapping matrices from the source and target domains to discover latent features for learning. However, these methods barely consider the reconstruction error to measure the information loss during the mapping procedure. In this paper, we propose to jointly capture the information and match the source and target domain distributions in the latent feature space. In the learning model, we propose to minimize the reconstruction loss between the original and reconstructed representations to preserve information during transformation and reduce the Maximum Mean Discrepancy between the source and target domains to align their distributions. The resulting minimization problem involves two projection variables with orthogonal constraints that can be solved by the generalized gradient flow method, which can preserve orthogonal constraints in the computational procedure. We conduct extensive experiments on several image classification datasets to demonstrate that the effectiveness and efficiency of the proposed method are better than those of state-of-the-art HDA methods.

Multi-Dimensional Visual Data Completion via Low-Rank Tensor Representation Under Coupled Transform.

This paper addresses the tensor completion problem, which aims to recover missing information of multi-dimensional images. How to represent a low-rank structure embedded in the underlying data is the key issue in tensor completion. In this work, we suggest a novel low-rank tensor representation based on coupled transform, which fully exploits the spatial multi-scale nature and redundancy in spatial and spectral/temporal dimensions, leading to a better low tensor multi-rank approximation. More precisely, this representation is achieved by using two-dimensional framelet transform for the two spatial dimensions, one/two-dimensional Fourier transform for the temporal/spectral dimension, and then Karhunen-Loéve transform (via singular value decomposition) for the transformed tensor. Based on this low-rank tensor representation, we formulate a novel low-rank tensor completion model for recovering missing information in multi-dimensional visual data, which leads to a convex optimization problem. To tackle the proposed model, we develop the alternating directional method of multipliers (ADMM) algorithm tailored for the structured optimization problem. Numerical examples on color images, multispectral images, and videos illustrate that the proposed method outperforms many state-of-the-art methods in qualitative and quantitative aspects.

Variational model for simultaneously image denoising and contrast enhancement.

The performance of contrast enhancement is degraded when input images are noisy. In this paper, we propose and develop a variational model for simultaneously image denoising and contrast enhancement. The idea is to propose a variational approach containing an energy functional to adjust the pixel values of an input image directly so that the resulting histogram can be redistributed to be uniform and the noise of the image can be removed. In the proposed model, a histogram equalization term is considered for image contrast enhancement, a total variational term is incorporate to remove the noise of the input image, and a fidelity term is added to keep the structure and the texture of the input image. The existence of the minimizer and the convergence of the proposed algorithm are studied and analyzed. Experimental results are presented to show the effectiveness of the proposed model compared with existing methods in terms of several measures: average local contrast, discrete entropy, structural similarity index, measure of enhancement, absolute measure of enhancement, and second derivative like measure of enhancement.

Structured Dictionary Learning for Image Denoising under Mixed Gaussian and Impulse Noise.

Although image denoising as a basic task of image restoration has been widely studied in the past decades, there are not many studies on mixed noise denoising. In this paper, we propose two structured dictionary learning models to recover images corrupted by mixed Gaussian and impulse noise. These two models can be merged as ℓp-norm fidelity plus ℓq-norm regularization. The fidelity term is used to fit image patches and the regularization term is employed for sparse coding. Particularly, we utilize proximal (and proximal linearized) alternating minimization methods as the main solvers to deal with these two models. We remove the Gaussian noise under the assumption that the uncorrupted image can be approximated with a linear representation under an appropriate orthogonal basis. We use different ways to remove impulse noise for these two models. The experimental results are reported to compare the existing methods and demonstrate the performance of the proposed denoising model is better than the other existing methods in terms of some quality assessment metrics.

Dimensionality reduction for single cell RNA sequencing data using constrained robust non-negative matrix factorization.

Single cell RNA-sequencing (scRNA-seq) technology, a powerful tool for analyzing the entire transcriptome at single cell level, is receiving increasing research attention. The presence of dropouts is an important characteristic of scRNA-seq data that may affect the performance of downstream analyses, such as dimensionality reduction and clustering. Cells sequenced to lower depths tend to have more dropouts than those sequenced to greater depths. In this study, we aimed to develop a dimensionality reduction method to address both dropouts and the non-negativity constraints in scRNA-seq data. The developed method simultaneously performs dimensionality reduction and dropout imputation under the non-negative matrix factorization (NMF) framework. The dropouts were modeled as a non-negative sparse matrix. Summation of the observed data matrix and dropout matrix was approximated by NMF. To ensure the sparsity pattern was maintained, a weighted ℓ1 penalty that took into account the dependency of dropouts on the sequencing depth in each cell was imposed. An efficient algorithm was developed to solve the proposed optimization problem. Experiments using both synthetic data and real data showed that dimensionality reduction via the proposed method afforded more robust clustering results compared with those obtained from the existing methods, and that dropout imputation improved the differential expression analysis.