A multivariate adaptive gradient algorithm with reduced tuning efforts.

Large neural networks usually perform well for executing machine learning tasks. However, models that achieve state-of-the-art performance involve arbitrarily large number of parameters and therefore their training is very expensive. It is thus desired to implement methods with small per-iteration costs, fast convergence rates, and reduced tuning. This paper proposes a multivariate adaptive gradient descent method that meets the above attributes. The proposed method updates every element of the model parameters separately in a computationally efficient manner using an adaptive vector-form learning rate, resulting in low per-iteration cost. The adaptive learning rate computes the absolute difference of current and previous model parameters over the difference in subgradients of current and previous state estimates. In the deterministic setting, we show that the cost function value converges at a linear rate for smooth and strongly convex cost functions. Whereas in both the deterministic and stochastic setting, we show that the gradient converges in expectation at the order of O(1/k) for a non-convex cost function with Lipschitz continuous gradient. In addition, we show that after T iterates, the cost function of the last iterate scales as O(log(T)/T) for non-smooth strongly convex cost functions. Effectiveness of the proposed method is validated on convex functions, smooth non-convex function, non-smooth convex function, and four image classification data sets, whilst showing that its execution requires hardly any tuning unlike existing popular optimizers that entail relatively large tuning efforts. Our empirical results show that our proposed algorithm provides the best overall performance when comparing it to tuned state-of-the-art optimizers.

Learning From Multiple Imperfect Instructors in Sensor Networks.

This paper presents a sequential learning framework for sensors in a network, where a few sensors assume the role of an instructor to train other sensors in the network. The instructors provide estimated labels for measurements of new sensors. These labels are possibly noisy, because a classifier of the instructor may not be perfect. A recursive density estimator is proposed to obtain the true measurement model (i.e., the observation density conditioned on the label) in spite of the training with noisy labels. Specifically, this paper answers the question "Can a sensor train other sensors?", provides necessary conditions for sensors to act as instructors, presents a sequential learning framework using recursive nonparametric kernel density estimation, and provides a convergence rate for the expected error in an observation density. The underlying concepts are illustrated and validated with simulation results.

Non-negative sparse autoencoder neural networks for the detection of overlapping, hierarchical communities in networked datasets.

We propose the first use of a non-negative sparse autoencoder (NNSAE) neural network for community structure detection in complex networks. The NNSAE learns a compressed representation of a set of fixed-length, weighted random walks over the network, and communities are detected as subsets of network nodes corresponding to non-negligible elements of the basis vectors of this compression. The NNSAE model is efficient and online. When utilized for community structure detection, it is able to uncover potentially overlapping and hierarchical community structure in large networks.

Probabilistic color matching and tracking of human subjects.

Pattern discovery algorithms based on the computational mechanics (CM) method have been shown to succinctly describe underlying patterns in data through the reconstruction of minimum probabilistic finite state automata (PFSA). We apply the CM approach toward the tracking of human subjects in real time by matching and tracking the underlying color pattern as observed from a fixed camera. Objects are extracted from a video sequence, and then raster scanned, decomposed with a one-dimensional Haar wavelet transform, and symbolized with the aid of a red-green-blue (RGB) color cube. The clustered causal state algorithm is then used to reconstruct the corresponding PFSA. Tracking is accomplished by generating the minimum PFSA for each subsequent frame, followed by matching the PFSAs to the previous frame. Results show that there is an optimum alphabet size and segmentation of the RGB color cube for efficient tracking.