Deep Learning Model Compression With Rank Reduction in Tensor Decomposition.

Large neural network models are hard to deploy on lightweight edge devices demanding large network bandwidth. In this article, we propose a novel deep learning (DL) model compression method. Specifically, we present a dual-model training strategy with an iterative and adaptive rank reduction (RR) in tensor decomposition. Our method regularizes the DL models while preserving model accuracy. With adaptive RR, the hyperparameter search space is significantly reduced. We provide a theoretical analysis of the convergence and complexity of the proposed method. Testing our method for the LeNet, VGG, ResNet, EfficientNet, and RevCol over MNIST, CIFAR-10/100, and ImageNet datasets, our method outperforms the baseline compression methods in both model compression and accuracy preservation. The experimental results validate our theoretical findings. For the VGG-16 on CIFAR-10 dataset, our compressed model has shown a 0.88% accuracy gain with 10.41 times storage reduction and 6.29 times speedup. For the ResNet-50 on ImageNet dataset, our compressed model results in 2.36 times storage reduction and 2.17 times speedup. In federated learning (FL) applications, our scheme reduces 13.96 times the communication overhead. In summary, our compressed DL method can improve the image understanding and pattern recognition processes significantly.

Long-range zero-shot generative deep network quantization.

Quantization approximates a deep network model with floating-point numbers by the model with low bit width numbers, thereby accelerating inference and reducing computation. Zero-shot quantization, which aims to quantize a model without access to the original data, can be achieved by fitting the real data distribution through data synthesis. However, it has been observed that zero-shot quantization leads to inferior performance compared to post-training quantization with real data for two primary reasons: 1) a normal generator has difficulty obtaining a high diversity of synthetic data since it lacks long-range information to allocate attention to global features, and 2) synthetic images aim to simulate the statistics of real data, which leads to weak intra-class heterogeneity and limited feature richness. To overcome these problems, we propose a novel deep network quantizer called long-range zero-shot generative deep network quantization (LRQ). Technically, we propose a long-range generator (LRG) to learn long-range information instead of simple local features. To incorporate more global features into the synthetic data, we use long-range attention with large-kernel convolution in the generator. In addition, we also present an adversarial margin add (AMA) module to force intra-class angular enlargement between the feature vector and class center. The AMA module forms an adversarial process that increases the convergence difficulty of the loss function, which is opposite to the training objective of the original loss function. Furthermore, to transfer knowledge from the full-precision network, we also utilize decoupled knowledge distillation. Extensive experiments demonstrate that LRQ obtains better performance than other competitors.

Low-Rank Tensor Completion Based on Self-Adaptive Learnable Transforms.

The tensor nuclear norm (TNN), defined as the sum of nuclear norms of frontal slices of the tensor in a frequency domain, has been found useful in solving low-rank tensor recovery problems. Existing TNN-based methods use either fixed or data-independent transformations, which may not be the optimal choices for the given tensors. As the consequence, these methods cannot exploit the potential low-rank structure of tensor data adaptively. In this article, we propose a framework called self-adaptive learnable transform (SALT) to learn a transformation matrix from the given tensor. Specifically, SALT aims to learn a lossless transformation that induces a lower average-rank tensor, where the Schatten- p quasi-norm is used as the rank proxy. Then, because SALT is less sensitive to the orientation, we generalize SALT to other dimensions of tensor (SALTS), namely, learning three self-adaptive transformation matrices simultaneously from given tensor. SALTS is able to adaptively exploit the potential low-rank structures in all directions. We provide a unified optimization framework based on alternating direction multiplier method for SALTS model and theoretically prove the weak convergence property of the proposed algorithm. Experimental results in hyperspectral image (HSI), color video, magnetic resonance imaging (MRI), and COIL-20 datasets show that SALTS is much more accurate in tensor completion than existing methods. The demo code can be found at https://faculty.uestc.edu.cn/gaobin/zh_ CN/lwcg/153392/list/index.htm.

The neurophysiology of the intervention strategies of Awareness Training Program on emotion regulation.

Emotion regulation is essential for healthy living. Previous studies have found that mental training such as compassion meditation could help with emotion regulation. However, the underlying neural mechanism and possible intervention strategies of group-based Mahayana Buddhist intervention involved in emotion regulation are still unclear. This event-related potential (ERP) study investigated how compassion and wisdom meditations, two key components of the Awareness Training Program (ATP), may regulate emotion during different mental processing stages, namely attention deployment, cognitive change, and response modification. Eighty-five middle-aged working adults with moderate stress were voluntarily recruited for this study, using a 128-channel electroencephalogram system. After 7 weeks of training, participants (ATP attendance, n = 42; waitlist control, n = 43) were instructed to view negative pictures while practicing compassion or wisdom meditation, with corresponding priming words. Another normal priming condition and a neutral picture condition were set as control conditions. ERP results in the ATP group showed that negative pictures induced greater prefrontal activity (N400 component) in both compassion and wisdom meditation conditions compared with the normal condition, while the control group showed little difference between the conditions. Significantly higher heart rate variability was found in the compassion but not wisdom meditation when compared with the neutral priming condition. Correspondent changes in behavioural data were also found. Converging evidence showed that compassion meditation training could modulate negative emotion processing in stages of attention deployment, cognitive change, and behavioural responses. The prefrontal lobe could play an important role in the process of emotion regulation by compassion meditation, possibly due to the emphasis of the ATP on contemplative practices.

Exactly Robust Kernel Principal Component Analysis.

Robust principal component analysis (RPCA) can recover low-rank matrices when they are corrupted by sparse noises. In practice, many matrices are, however, of high rank and, hence, cannot be recovered by RPCA. We propose a novel method called robust kernel principal component analysis (RKPCA) to decompose a partially corrupted matrix as a sparse matrix plus a high- or full-rank matrix with low latent dimensionality. RKPCA can be applied to many problems such as noise removal and subspace clustering and is still the only unsupervised nonlinear method robust to sparse noises. Our theoretical analysis shows that, with high probability, RKPCA can provide high recovery accuracy. The optimization of RKPCA involves nonconvex and indifferentiable problems. We propose two nonconvex optimization algorithms for RKPCA. They are alternating direction method of multipliers with backtracking line search and proximal linearized minimization with adaptive step size (AdSS). Comparative studies in noise removal and robust subspace clustering corroborate the effectiveness and the superiority of RKPCA.

Accelerated low-rank representation for subspace clustering and semi-supervised classification on large-scale data.

The scalability of low-rank representation (LRR) to large-scale data is still a major research issue, because it is extremely time-consuming to solve singular value decomposition (SVD) in each optimization iteration especially for large matrices. Several methods were proposed to speed up LRR, but they are still computationally heavy, and the overall representation results were also found degenerated. In this paper, a novel method, called accelerated LRR (ALRR) is proposed for large-scale data. The proposed accelerated method integrates matrix factorization with nuclear-norm minimization to find a low-rank representation. In our proposed method, the large square matrix of representation coefficients is transformed into a significantly smaller square matrix, on which SVD can be efficiently implemented. The size of the transformed matrix is not related to the number of data points and the optimization of ALRR is linear with the number of data points. The proposed ALRR is convex, accurate, robust, and efficient for large-scale data. In this paper, ALRR is compared with state-of-the-art in subspace clustering and semi-supervised classification on real image datasets. The obtained results verify the effectiveness and superiority of the proposed ALRR method.

Matrix completion by deep matrix factorization.

Conventional methods of matrix completion are linear methods that are not effective in handling data of nonlinear structures. Recently a few researchers attempted to incorporate nonlinear techniques into matrix completion but there still exists considerable limitations. In this paper, a novel method called deep matrix factorization (DMF) is proposed for nonlinear matrix completion. Different from conventional matrix completion methods that are based on linear latent variable models, DMF is on the basis of a nonlinear latent variable model. DMF is formulated as a deep-structure neural network, in which the inputs are the low-dimensional unknown latent variables and the outputs are the partially observed variables. In DMF, the inputs and the parameters of the multilayer neural network are simultaneously optimized to minimize the reconstruction errors for the observed entries. Then the missing entries can be readily recovered by propagating the latent variables to the output layer. DMF is compared with state-of-the-art methods of linear and nonlinear matrix completion in the tasks of toy matrix completion, image inpainting and collaborative filtering. The experimental results verify that DMF is able to provide higher matrix completion accuracy than existing methods do and DMF is applicable to large matrices.

Sparse subspace clustering for data with missing entries and high-rank matrix completion.

Many methods have recently been proposed for subspace clustering, but they are often unable to handle incomplete data because of missing entries. Using matrix completion methods to recover missing entries is a common way to solve the problem. Conventional matrix completion methods require that the matrix should be of low-rank intrinsically, but most matrices are of high-rank or even full-rank in practice, especially when the number of subspaces is large. In this paper, a new method called Sparse Representation with Missing Entries and Matrix Completion is proposed to solve the problems of incomplete-data subspace clustering and high-rank matrix completion. The proposed algorithm alternately computes the matrix of sparse representation coefficients and recovers the missing entries of a data matrix. The proposed algorithm recovers missing entries through minimizing the representation coefficients, representation errors, and matrix rank. Thorough experimental study and comparative analysis based on synthetic data and natural images were conducted. The presented results demonstrate that the proposed algorithm is more effective in subspace clustering and matrix completion compared with other existing methods.

Using Wavelet Entropy to Demonstrate how Mindfulness Practice Increases Coordination between Irregular Cerebral and Cardiac Activities.

In both the East and West, traditional teachings say that the mind and heart are somehow closely correlated, especially during spiritual practice. One difficulty in proving this objectively is that the natures of brain and heart activities are quite different. In this paper, we propose a methodology that uses wavelet entropy to measure the chaotic levels of both electroencephalogram (EEG) and electrocardiogram (ECG) data and show how this may be used to explore the potential coordination between the mind and heart under different experimental conditions. Furthermore, Statistical Parametric Mapping (SPM) was used to identify the brain regions in which the EEG wavelet entropy was the most affected by the experimental conditions. As an illustration, the EEG and ECG were recorded under two different conditions (normal rest and mindful breathing) at the beginning of an 8-week standard Mindfulness-based Stress Reduction (MBSR) training course (pretest) and after the course (posttest). Using the proposed method, the results consistently showed that the wavelet entropy of the brain EEG decreased during the MBSR mindful breathing state as compared to that during the closed-eye resting state. Similarly, a lower wavelet entropy of heartrate was found during MBSR mindful breathing. However, no difference in wavelet entropy during MBSR mindful breathing was found between the pretest and posttest. No correlation was observed between the entropy of brain waves and the entropy of heartrate during normal rest in all participants, whereas a significant correlation was observed during MBSR mindful breathing. Additionally, the most well-correlated brain regions were located in the central areas of the brain. This study provides a methodology for the establishment of evidence that mindfulness practice (i.e., mindful breathing) may increase the coordination between mind and heart activities.

Entrainment of chaotic activities in brain and heart during MBSR mindfulness training.

The activities of the brain and the heart are dynamic, chaotic, and possibly intrinsically coordinated. This study aims to investigate the effect of Mindfulness-Based Stress Reduction (MBSR) program on the chaoticity of electronic activities of the brain and the heart, and to explore their potential correlation. Electroencephalogram (EEG) and electrocardiogram (ECG) were recorded at the beginning of an 8-week standard MBSR training course and after the course. EEG spectrum analysis was carried out, wavelet entropies (WE) of EEG (together with reconstructed cortical sources) and heart rate were calculated, and their correlation was investigated. We found enhancement of EEG power of alpha and beta waves and lowering of delta waves power during MBSR training state as compared to normal resting state. Wavelet entropy analysis indicated that MBSR mindfulness meditation could reduce the chaotic activities of both EEG and heart rate as a change of state. However, longitudinal change of trait may need more long-term training. For the first time, our data demonstrated that the chaotic activities of the brain and the heart became more coordinated during MBSR training, suggesting that mindfulness training may increase the entrainment between mind and body. The 3D brain regions involved in the change in mental states were identified.

Repetitive Religious Chanting Modulates the Late-Stage Brain Response to Fear- and Stress-Provoking Pictures.

Chanting and praying are among the most popular religious activities, which are said to be able to alleviate people's negative emotions. However, the neural mechanisms underlying this mental exercise and its temporal course have hardly been investigated. Here, we used event-related potentials (ERPs) to explore the effects of chanting the name of a Buddha (Amitabha) on the brain's response to viewing negative pictures that were fear- and stress-provoking. We recorded and analyzed electroencephalography (EEG) data from 21 Buddhists with chanting experience as they viewed negative and neutral pictures. Participants were instructed to chant the names of Amitabha or Santa Claus silently to themselves or simply remain silent (no-chanting condition) during picture viewing. To measure the physiological changes corresponding to negative emotions, electrocardiogram and galvanic skin response data were also collected. Results showed that viewing negative pictures (vs. neutral pictures) increased the amplitude of the N1 component in all the chanting conditions. The amplitude of late positive potential (LPP) also increased when the negative pictures were viewed under the no-chanting and the Santa Claus condition. However, increased LPP was not observed when chanting Amitabha. The ERP source analysis confirmed this finding and showed that increased LPP mainly originated from the central-parietal regions of the brain. In addition, the participants' heart rates decreased significantly when viewing negative pictures in the Santa Claus condition. The no-chanting condition had a similar decreasing trend although not significant. However, while chanting Amitabha and viewing negative pictures participants' heart rate did not differ significantly from that observed during neutral picture viewing. It is possible that the chanting of Amitabha might have helped the participants to develop a religious schema and neutralized the effect of the negative stimuli. These findings echo similar research findings on Christian religious practices and brain responses to negative stimuli. Hence, prayer/religious practices may have cross-cultural universality in emotion regulation. This study shows for the first time that Buddhist chanting, or in a broader sense, repetition of religious prayers will not modulate brain responses to negative stimuli during the early perceptual stage, but only during the late-stage emotional/cognitive processing.