Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach.

Universal approximation capability, also referred to as universality, is an important property of deep neural networks, endowing them with the potency to accurately represent the underlying target function in learning tasks. In practice, the architecture of deep neural networks largely influences the performance of the models. However, most existing methodologies for designing neural architectures, such as the heuristic manual design or neural architecture search, ignore the universal approximation property, thus losing a potential safeguard about the performance. In this paper, we propose a unified framework to design the architectures of deep neural networks with a universality guarantee based on first-order optimization algorithms, where the forward pass is interpreted as the updates of an optimization algorithm. The (explicit or implicit) network is designed by replacing each gradient term in the algorithm with a learnable module similar to a two-layer network or its derivatives Specifically, we explore the realm of width-bounded neural networks, a common practical scenario, showcasing their universality. Moreover, adding operations of normalization, downsampling, and upsampling does not hurt the universality. To the best of our knowledge, this is the first work that width-bounded networks with universal approximation guarantee can be designed in a principled way. Our framework can inspire a variety of neural architectures including some renowned structures such as ResNet and DenseNet, as well as novel innovations. The experimental results on image classification problems demonstrate that the newly inspired networks are competitive and surpass the baselines of ResNet, DenseNet, as well as the advanced ConvNeXt and ViT, testifying to the effectiveness of our framework.

Sampling complex topology structures for spiking neural networks.

Spiking Neural Networks (SNNs) have been considered a potential competitor to Artificial Neural Networks (ANNs) due to their high biological plausibility and energy efficiency. However, the architecture design of SNN has not been well studied. Previous studies either use ANN architectures or directly search for SNN architectures under a highly constrained search space. In this paper, we aim to introduce much more complex connection topologies to SNNs to further exploit the potential of SNN architectures. To this end, we propose the topology-aware search space, which is the first search space that enables a more diverse and flexible design for both the spatial and temporal topology of the SNN architecture. Then, to efficiently obtain architecture from our search space, we propose the spatio-temporal topology sampling (STTS) algorithm. By leveraging the benefits of random sampling, STTS can yield powerful architecture without the need for an exhaustive search process, making it significantly more efficient than alternative search strategies. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate the effectiveness of our method. Notably, we obtain 70.79% top-1 accuracy on ImageNet with only 4 time steps, 1.79% higher than the second best model. Our code is available under https://github.com/stiger1000/Random-Sampling-SNN.

Multidimensional Data Processing With Bayesian Inference via Structural Block Decomposition.

How to handle large multidimensional datasets, such as hyperspectral images and video information, efficiently and effectively plays a critical role in big-data processing. The characteristics of low-rank tensor decomposition in recent years demonstrate the essentials in describing the tensor rank, which often leads to promising approaches. However, most current tensor decomposition models consider the rank-1 component simply to be the vector outer product, which may not fully capture the correlated spatial information effectively for large-scale and high-order multidimensional datasets. In this article, we develop a new novel tensor decomposition model by extending it to the matrix outer product or called Bhattacharya-Mesner product, to form an effective dataset decomposition. The fundamental idea is to decompose tensors structurally in a compact manner as much as possible while retaining data spatial characteristics in a tractable way. By incorporating the framework of the Bayesian inference, a new tensor decomposition model on the subtle matrix unfolding outer product is established for both tensor completion and robust principal component analysis problems, including hyperspectral image completion and denoising, traffic data imputation, and video background subtraction. Numerical experiments on real-world datasets demonstrate the highly desirable effectiveness of the proposed approach.

Metafeature Selection via Multivariate Sparse-Group Lasso Learning for Automatic Hyperparameter Configuration Recommendation.

The performance of classification algorithms is mainly governed by the hyperparameter settings deployed in applications, and the search for desirable hyperparameter configurations usually is quite challenging due to the complexity of datasets. Metafeatures are a group of measures that characterize the underlying dataset from various aspects, and the corresponding recommendation algorithm fully relies on the appropriate selection of metafeatures. Metalearning (MtL), aiming to improve the learning algorithm itself, requires development in integrating features, models, and algorithm learning to accomplish its goal. In this article, we develop a multivariate sparse-group Lasso (SGLasso) model embedded with MtL capacity in recommending suitable configurations via learning. The main idea is to select the principal metafeatures by removing those redundant or irregular ones, promoting both efficiency and performance in the hyperparameter configuration recommendation. To be specific, we first extract the metafeatures and classification performance of a set of configurations from the collection of historical datasets, and then, a metaregression task is established through SGLasso to capture the main characteristics of the underlying relationship between metafeatures and historical performance. For a new dataset, the classification performance of configurations can be estimated through the selected metafeatures so that the configuration with the highest predictive performance in terms of the new dataset can be generated. Furthermore, a general MtL architecture combined with our model is developed. Extensive experiments are conducted on 136 UCI datasets, demonstrating the effectiveness of the proposed approach. The empirical results on the well-known SVM show that our model can effectively recommend suitable configurations and outperform the existing MtL-based methods and the well-known search-based algorithms, such as random search, Bayesian optimization, and Hyperband.

Bayesian Dictionary Learning on Robust Tubal Transformed Tensor Factorization.

The recent study on tensor singular value decomposition (t-SVD) that performs the Fourier transform on the tubes of a third-order tensor has gained promising performance on multidimensional data recovery problems. However, such a fixed transformation, e.g., discrete Fourier transform and discrete cosine transform, lacks being self-adapted to the change of different datasets, and thus, it is not flexible enough to exploit the low-rank and sparse property of the variety of multidimensional datasets. In this article, we consider a tube as an atom of a third-order tensor and construct a data-driven learning dictionary from the observed noisy data along the tubes of the given tensor. Then, a Bayesian dictionary learning (DL) model with tensor tubal transformed factorization, aiming to identify the underlying low-tubal-rank structure of the tensor effectively via the data-adaptive dictionary, is developed to solve the tensor robust principal component analysis (TRPCA) problem. With the defined pagewise tensor operators, a variational Bayesian DL algorithm is established and updates the posterior distributions instantaneously along the third dimension to solve the TPRCA. Extensive experiments on real-world applications, such as color image and hyperspectral image denoising and background/foreground separation problems, demonstrate both effectiveness and efficiency of the proposed approach in terms of various standard metrics.

SPIDE: A purely spike-based method for training feedback spiking neural networks.

Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. However, most supervised SNN training methods, such as conversion from artificial neural networks or direct training with surrogate gradients, require complex computation rather than spike-based operations of spiking neurons during training. In this paper, we study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method, implicit differentiation on the equilibrium state (IDE), for supervised learning with purely spike-based computation, which demonstrates the potential for energy-efficient training of SNNs. Specifically, we introduce ternary spiking neuron couples and prove that implicit differentiation can be solved by spikes based on this design, so the whole training procedure, including both forward and backward passes, is made as event-driven spike computation, and weights are updated locally with two-stage average firing rates. Then we propose to modify the reset membrane potential to reduce the approximation error of spikes. With these key components, we can train SNNs with flexible structures in a small number of time steps and with firing sparsity during training, and the theoretical estimation of energy costs demonstrates the potential for high efficiency. Meanwhile, experiments show that even with these constraints, our trained models can still achieve competitive results on MNIST, CIFAR-10, CIFAR-100, and CIFAR10-DVS.

Tensorial Multiview Representation for Saliency Detection via Nonconvex Approach.

In the study of salient object detection, multiview features play an important role in identifying various underlying salient objects. As to current common patch-based methods, all different features are handled directly by stacking them into a high-dimensional vector to represent related image patches. These approaches ignore the correlations inhering in the original spatial structure, which may lead to the loss of certain underlying characterization such as view interaction. In this article, different from currently available approaches, a tensorial feature representation framework is developed for the salient object detection in order to better explore the complementary information of multiview features. Under the tensor framework, a tensor low-rank constraint is applied to the background to capture its intrinsic structure, a tensor group sparsity regularization is posed on the salient part, and a tensorial sliced Laplacian regularization is then introduced to enlarge the gap between the subspaces of the background and salient object. Moreover, a nonconvex tensor Log-determinant function, instead of the tensor nuclear norm, is adopted to approximate the tensor rank for effectively suppressing the confusing information resulted from underlying complex backgrounds. Further, we have deduced the closed-form solution of this nonconvex minimization problem and established a feasible algorithm whose convergence is mathematically proven. Experiments on five well-known public datasets are provided and the simulations demonstrate that our method outperforms the latest unsupervised handcrafted features-based methods in the literature. Furthermore, our model is flexible with various deep features and is competitive with the state-of-the-art approaches.

Unsupervised Learning for Salient Object Detection via Minimization of Bilinear Factor Matrix Norm.

Saliency detection is an important but challenging task in the study of computer vision. In this article, we develop a new unsupervised learning approach for the saliency detection by an intrinsic regularization model, in which the Schatten-2/3 norm is integrated with the nonconvex sparse l2/3 norm. The l2/3 -norm is shown to be capable of detecting consistent values among sparse foreground by using image geometrical structure and feature similarity, while the Schatten-2/3 norm can capture the lower rank of background by matrix factorization. To improve effective performance of separation for Schatten-2/3-norm and l2/3 -norm, a Laplacian regularization is adopted to the foreground for the smoothness. The proposed model essentially converts the required nonconvex optimization problem into the convex one, conducted by splitting the objective function based on singular value decomposition on one much smaller factor matrix and then optimized by using the alternating direction method of the multiplier. The convergence of the proposed algorithm is discussed in detail. Extensive experiments on three benchmark datasets demonstrate that our unsupervised learning approach is very competitive and appears to be more consistent across various salient objects than the current existing approaches.

A New Automatic Hyperparameter Recommendation Approach Under Low-Rank Tensor Completion e Framework.

Hyperparameter optimization (HPO), characterized by hyperparameter tuning, is not only a critical step for effective modeling but also is the most time-consuming process in machine learning. Traditional search-based algorithms tend to require extensive configuration evaluations for each round to select the desirable hyperparameters during the process, and they are often very inefficient for the implementations on large-scale tasks. In this paper, we study the HPO problem via meta-learning (MtL) approach under the low-rank tensor completion (LRTC) framework. Our proposed approach predicts the performance for hyperparameters of new problems based on their previous performance so that the underlying suitable hyperparameters with better efficiency can be attained. Different from existing approaches, the hyperparameter performance space is instantiated under tensor framework that can preserve the spatial structure and reflect the correlations among the adjacent hyperparameters. When some partial evaluations are available for a new problem, the task of estimating the performance of the unevaluated hyperparameters can be formulated as a tensor completion (TC) problem. Toward the completion purpose, we develop an LRTC algorithm utilizing the sum of nuclear norm (SNN) model. A kernelized version is further developed to capture the nonlinear structure of the performance space. In addition, a corresponding coupled matrix factorization (CMF) algorithm is established to render the predictions solely depend on the meta-features to avoid additional hyperparameter evaluations. Finally, a strategy integrating LRTC and CMF is provided to further enhance the recommendation capacity. We test recommendation performance with our proposed methods for classical SVM and the state-of-the-art deep neural networks such as vision transformer (ViT) and residual network (ResNet), and the obtained results demonstrate the effectiveness of our approaches under various evaluation metrics by comparing with the baselines commonly used for MtL.

Efficient and generalizable cross-patient epileptic seizure detection through a spiking neural network.

Introduction: Epilepsy is a global chronic disease that brings pain and inconvenience to patients, and an electroencephalogram (EEG) is the main analytical tool. For clinical aid that can be applied to any patient, an automatic cross-patient epilepsy seizure detection algorithm is of great significance. Spiking neural networks (SNNs) are modeled on biological neurons and are energy-efficient on neuromorphic hardware, which can be expected to better handle brain signals and benefit real-world, low-power applications. However, automatic epilepsy seizure detection rarely considers SNNs. Methods: In this article, we have explored SNNs for cross-patient seizure detection and discovered that SNNs can achieve comparable state-of-the-art performance or a performance that is even better than artificial neural networks (ANNs). We propose an EEG-based spiking neural network (EESNN) with a recurrent spiking convolution structure, which may better take advantage of temporal and biological characteristics in EEG signals. Results: We extensively evaluate the performance of different SNN structures, training methods, and time settings, which builds a solid basis for understanding and evaluation of SNNs in seizure detection. Moreover, we show that our EESNN model can achieve energy reduction by several orders of magnitude compared with ANNs according to the theoretical estimation. Discussion: These results show the potential for building high-performance, low-power neuromorphic systems for seizure detection and also broaden real-world application scenarios of SNNs.

Training much deeper spiking neural networks with a small number of time-steps.

Spiking Neural Network (SNN) is a promising energy-efficient neural architecture when implemented on neuromorphic hardware. The Artificial Neural Network (ANN) to SNN conversion method, which is the most effective SNN training method, has successfully converted moderately deep ANNs to SNNs with satisfactory performance. However, this method requires a large number of time-steps, which hurts the energy efficiency of SNNs. How to effectively covert a very deep ANN (e.g., more than 100 layers) to an SNN with a small number of time-steps remains a difficult task. To tackle this challenge, this paper makes the first attempt to propose a novel error analysis framework that takes both the "quantization error" and the "deviation error" into account, which comes from the discretization of SNN dynamicsthe neuron's coding scheme and the inconstant input currents at intermediate layers, respectively. Particularly, our theories reveal that the "deviation error" depends on both the spike threshold and the input variance. Based on our theoretical analysis, we further propose the Threshold Tuning and Residual Block Restructuring (TTRBR) method that can convert very deep ANNs (>100 layers) to SNNs with negligible accuracy degradation while requiring only a small number of time-steps. With very deep networks, our TTRBR method achieves state-of-the-art (SOTA) performance on the CIFAR-10, CIFAR-100, and ImageNet classification tasks.

Training Neural Networks by Lifted Proximal Operator Machines.

We present the lifted proximal operator machine (LPOM) to train fully-connected feed-forward neural networks. LPOM represents the activation function as an equivalent proximal operator and adds the proximal operators to the objective function of a network as penalties. LPOM is block multi-convex in all layer-wise weights and activations. This allows us to develop a new block coordinate descent (BCD) method with convergence guarantee to solve it. Due to the novel formulation and solving method, LPOM only uses the activation function itself and does not require any gradient steps. Thus it avoids the gradient vanishing or exploding issues, which are often blamed in gradient-based methods. Also, it can handle various non-decreasing Lipschitz continuous activation functions. Additionally, LPOM is almost as memory-efficient as stochastic gradient descent and its parameter tuning is relatively easy. We further implement and analyze the parallel solution of LPOM. We first propose a general asynchronous-parallel BCD method with convergence guarantee. Then we use it to solve LPOM, resulting in asynchronous-parallel LPOM. For faster speed, we develop the synchronous-parallel LPOM. We validate the advantages of LPOM on various network architectures and datasets. We also apply synchronous-parallel LPOM to autoencoder training and demonstrate its fast convergence and superior performance.

Synchronization for stochastic coupled networks with Lévy noise via event-triggered control.

This paper addresses the realization of almost sure synchronization problem for a new array of stochastic networks associated with delay and Lévy noise via event-triggered control. The coupling structure of the network is governed by a continuous-time homogeneous Markov chain. The nodes in the networks communicate with each other and update their information only at discrete-time instants so that the network workload can be minimized. Under the framework of stochastic process including Markov chain and Lévy process, and the convergence theorem of non-negative semi-martingales, we show that the Markovian coupled networks can achieve the almost sure synchronization by event-triggered control methodology. The results are further extended to the directed topology, where the coupling structure can be asymmetric. Furthermore, we also proved that the Zeno behavior can be excluded under our proposed approach, indicating that our framework is practically feasible. Numerical simulations are provided to demonstrate the effectiveness of the obtained theoretical results.

Centralized/decentralized event-triggered pinning synchronization of stochastic coupled networks with noise and incomplete transitional rate.

This paper studies the synchronous problem of Markovian switching complex networks associated with partly unknown transitional rates, stochastic noise, and randomly coupling strength. In order to achieve the synchronization for these array networks, event-triggered pinning control is established and developed, in which the pinning node undergoes a self-adapted switch, governed by a Markov chain. Two types of event-triggered sampling controls, centralized and decentralized event-triggered sampling, respectively, are established. Sufficient conditions for synchronization are developed by constructing a desirable stochastic Lyapunov functional as well as by employing the properties of Markov chain and Itoˆ integration. Numerical simulations are provided to demonstrate the effectiveness of the proposed approach.

A new development of non-local image denoising using fixed-point iteration for non-convex ℓp sparse optimization.

We proposed a new efficient image denoising scheme, which mainly leads to four important contributions whose approaches are different from existing ones. The first is to show the equivalence between the group-based sparse representation and the Schatten-p norm minimization problem, so that the sparsity of the coefficients for each group can be measured by estimating the underlying singular values. The second is that we construct the proximal operator for sparse optimization in ℓp space with p ∈ (0, 1] by using fixed-point iteration and obtained a new solution of Schatten-p norm minimization problem, which is more rigorous and accurate than current available results. The third is that we analyze the suitable setting of power p for each noise level σ = 20, 30, 50, 60, 75, 100, respectively. We find that the optimal value of p is inversely proportional to the noise level except for high level of noise, where the best values of p are 1 and 0.95, when the noise levels are respectively 75 and 100. Last we measure the structural similarity between two image patches and extends previous deterministic annealing-based solution to sparsity optimization problem through incorporating the idea of dictionary learning. Experimental results demonstrate that for every given noise level, the proposed Spatially Adaptive Fixed Point Iteration (SAFPI) algorithm attains the best denoising performance on the value of Peak Signal-to-Noise Ratio (PSNR) and structure similarity (SSIM), being able to retain the image structure information, which outperforms many state-of-the-art denoising methods such as Block-matching and 3D filtering (BM3D), Weighted Nuclear Norm Minimization (WNNM) and Weighted Schatten p-Norm Minimization (WSNM).

Synchronization of Nonlinearly and Stochastically Coupled Markovian Switching Networks via Event-Triggered Sampling.

This paper studies the exponential synchronization problem for a new array of nonlinearly and stochastically coupled networks via events-triggered sampling (ETS) by self-adaptive learning. The networks include the following features: 1) a Bernoulli stochastic variable is introduced to describe the random structural coupling; 2) a stochastic variable with positive mean is used to model the coupling strength; and 3) a continuous time homogeneous Markov chain is employed to characterize the dynamical switching of the coupling structure and pinned node sets. The proposed network model is capable to capture various stochastic effect of an external environment during the network operations. In order to reduce networks' workload, different ETS strategies for network self-adaptive learning are proposed under continuous and discrete monitoring, respectively. Based on these ETS approaches, several sufficient conditions for synchronization are derived by employing stochastic Lyapunov-Krasovskii functions, the properties of stochastic processes, and some linear matrix inequalities. Numerical simulations are provided to demonstrate the effectiveness of the theoretical results and the superiority of the proposed ETS approach.

A Simple Chaotic Map-Based Image Encryption System Using Both Plaintext Related Permutation and Diffusion.

Recently, to conquer most non-plain related chaos-based image cryptosystems' security flaws that cannot resist the powerful chosen/knownn plain-text attacks or differential attacks efficiently for less plaintext sensitivity, many plain related chaos-based image cryptosystems have been developed. Most cryptosystems that have adopted the traditional permutation-diffusion structure still have some drawbacks and security flaws: (1) most plaintext related image encryption schemes using only plaintext related confusion operation or only plaintext related diffusion operation relate to plaintext inadequately that cannot achieve high plaintext sensitivity; (2) in some algorithms, the generation of security key that needs to be sent to the receiver is determined by the original image, so these algorithms may not applicable to real-time image encryption; (3) most plaintext related image encryption schemes have less efficiency because more than one round permutation-diffusion operation is required to achieve high security. To obtain high security and efficiency, a simple chaotic based color image encryption system by using both plaintext related permutation and diffusion is presented in this paper. In our cryptosystem, the values of the parameters of cat map used in permutation stage are related to plain image and the parameters of cat map are also influenced by the diffusion operation. Thus, both the permutation stage and diffusion stage are related to plain images, which can obtain high key sensitivity and plaintext sensitivity to resist chosen/known plaintext attacks or differential attacks efficiently. Furthermore, only one round of plaintext related permutation and diffusion operation is performed to process the original image to obtain cipher image. Thus, the proposed scheme has high efficiency. Complete simulations are given and the simulation results prove the excellent security and efficiency of the proposed scheme.

Variable prey development time suppresses predator-prey cycles and enhances stability.

Although theoretical models have demonstrated that predator-prey population dynamics can depend critically on age (stage) structure and the duration and variability in development times of different life stages, experimental support for this theory is non-existent. We conducted an experiment with a host-parasitoid system to test the prediction that increased variability in the development time of the vulnerable host stage can promote interaction stability. Host-parasitoid microcosms were subjected to two treatments: Normal and High variance in the duration of the vulnerable host stage. In control and Normal-variance microcosms, hosts and parasitoids exhibited distinct population cycles. In contrast, insect abundances were 18-24% less variable in High- than Normal-variance microcosms. More significantly, periodicity in host-parasitoid population dynamics disappeared in the High-variance microcosms. Simulation models confirmed that stability in High-variance microcosms was sufficient to prevent extinction. We conclude that developmental variability is critical to predator-prey population dynamics and could be exploited in pest-management programs.

Synchronization of neural networks with stochastic perturbation via aperiodically intermittent control.

In this paper, the synchronization problem for neural networks with stochastic perturbation is studied with intermittent control via adaptive aperiodicity. Under the framework of stochastic theory and Lyapunov stability method, we develop some techniques of intermittent control with adaptive aperiodicity to achieve the synchronization of a class of neural networks, modeled by stochastic systems. Some effective sufficient conditions are established for the realization of synchronization of the underlying network. Numerical simulations of two examples are provided to illustrate the theoretical results obtained in the paper.

Estimation of the diffusion rate and crossing probability for biased edge movement between two different types of habitat.

One of the fundamental goals of ecology is to examine how dispersal affects the distribution and dynamics of insects across natural landscapes. These landscapes are frequently divided into patches of habitat embedded in a matrix of several non-habitat regions, and dispersal behavior could vary within each landscape element as well as the edges between elements. Reaction-diffusion models are a common way of modeling dispersal and species interactions in such landscapes, but to apply these models we also need methods of estimating the diffusion rate and any edge behavior parameters. In this paper, we present a method of estimating the diffusion rate using the mean occupancy time for a circular region. We also use mean occupancy time to estimate a parameter (the crossing probability) that governs one type of edge behavior often used in these models, a biased random walk. These new methods have some advantages over other methods of estimating these parameters, including reduced computational cost and ease of use in the field. They also provide a method of estimating the diffusion rate for a particular location in space, compared to existing methods that represent averages over large areas. We further examine the statistical properties of the new method through simulation, and discuss how mean occupancy time could be estimated in field experiments.