TRUST-TECH-Based Systematic Search for Multiple Local Optima in Deep Neural Nets.

Training deep neural networks (DNNs) rested heavily on efficient local solvers. Due to their local property, local solvers are sensitive to initialization and hyperparameters. In this article, a systematical method for finding multiple high-quality local optimal DNNs, based on the transformation under stability-retaining equilibria characterization (TRUST-TECH) method, is introduced. Our goal is to systematically search for multiple local optimal parameters for large models, such as DNNs, trained on large datasets. To achieve this, a dynamic searching path (DSP) method is proposed to provide improved search guidance used in TRUST-TECH. By integrating the DSP method with the TRUST-TECH (DSP-TT) method, multiple optimal training solutions with higher quality than randomly initialized ones can be obtained. To take advantage of these optimal solutions, a DSP-TT ensemble method is further developed. Experiments on various test cases show that the proposed DSP-TT method achieves considerable improvement over other ensemble methods developed for deep architectures. The DSP-TT ensemble method also shows diversity advantages over other ensemble methods.

Trust-Tech Source-Point Method for Systematically Computing Multiple Local Optimal Solutions: Theory and Method.

In this article, a Trust-Tech source-point method is proposed to systematically compute multiple local optimal solutions (LOSs) for continuous unconstrained nonlinear optimization problems. This proposed method consists of four stages. Stage I finds one LOS (in which existing effective optimizers can be applied), stage II is the stage of escaping an LOS while stage III is the stage for entering the stability region (SR) of another stable equilibrium point (SEP) (i.e., another LOS). Stage IV computes other SEPs (i.e., LOSs) in corresponding SRs. A theoretical foundation for both stages II and III is developed, and these theoretical results are quite general on their own. The proposed method is numerically evaluated to compute multiple LOSs. For instance, a total of 5085 LOSs have been computed by the proposed Trust-Tech source point method on a 50-D test function. In addition, the proposed method can find the global optimal solutions of several test functions with 50 dimensions and 100 dimensions.

Constrained Multiobjective Nonlinear Optimization: A User Preference Enabling Method.

A novel user preference enabling (UPE) method is developed to solve general constrained nonlinear multiple objective optimization (MOO) problems. User wish lists on the preferred range of each objective function are introduced and incorporated into the MOO formulation to form a user-preferred (UP) MOO problem. A theoretical foundation of the UP feasible region of MOO problems is developed. The developed theoretical work leads to the development of a UPE method for effectively solving the UP-MOO problems. Distinguishing features of the proposed method include its ability to compute a targeted Pareto solution and to serve as a complement to existing MOO methods, in the sense that the proposed method assists the existing MOO methods in computing the Pareto front by providing feasible solutions and UP feasible solutions. Both proposed UPE method and derived theoretical developments are evaluated on several test systems with promising results.

TRUST-TECH-based expectation maximization for learning finite mixture models.

In spite of the initialization problem, the Expectation-Maximization (EM) algorithm is widely used for estimating the parameters of finite mixture models. Most popular model-based clustering techniques might yield poor clusters if the parameters are not initialized properly. To reduce the sensitivity of initial points, a novel algorithm for learning mixture models from multivariate data is introduced in this paper. The proposed algorithm takes advantage of TRUST-TECH (TRansformation Under STability-reTaining Equilibra CHaracterization) to compute neighborhood local maxima on likelihood surface using stability regions. Basically, our method coalesces the advantages of the traditional EM with that of the dynamic and geometric characteristics of the stability regions of the corresponding nonlinear dynamical system of the log-likelihood function. Two phases namely, the EM phase and the stability region phase, are repeated alternatively in the parameter space to achieve improvements in the maximum likelihood. The EM phase obtains the local maximum of the likelihood function and the stability region phase helps to escape out of the local maximum by moving towards the neighboring stability regions. The algorithm has been tested on both synthetic and real datasets and the improvements in the performance compared to other approaches are demonstrated. The robustness with respect to initialization is also illustrated experimentally.

A Novel Consensus-Based Particle Swarm Optimization-Assisted Trust-Tech Methodology for Large-Scale Global Optimization.

A novel three-stage methodology, termed the "consensus-based particle swarm optimization (PSO)-assisted Trust-Tech methodology," to find global optimal solutions for nonlinear optimization problems is presented. It is composed of Trust-Tech methods, consensus-based PSO, and local optimization methods that are integrated to compute a set of high-quality local optimal solutions that can contain the global optimal solution. The proposed methodology compares very favorably with several recently developed PSO algorithms based on a set of small-dimension benchmark optimization problems and 20 large-dimension test functions from the CEC 2010 competition. The analytical basis for the proposed methodology is also provided. Experimental results demonstrate that the proposed methodology can rapidly obtain high-quality optimal solutions that can contain the global optimal solution. The scalability of the proposed methodology is promising.

A stability boundary based method for finding saddle points on potential energy surfaces.

The task of finding saddle points on potential energy surfaces plays a crucial role in understanding the dynamics of a micromolecule as well as in studying the folding pathways of macromolecules like proteins. The problem of finding the saddle points on a high dimensional potential energy surface is transformed into the problem of finding decomposition points of its corresponding nonlinear dynamical system. This paper introduces a new method based on TRUST-TECH (TRansformation Under STability reTained Equilibria CHaracterization) to compute saddle points on potential energy surfaces using stability boundaries. Our method explores the dynamic and geometric characteristics of stability boundaries of a nonlinear dynamical system. A novel trajectory adjustment procedure is used to trace the stability boundary. Our method was successful in finding the saddle points on different potential energy surfaces of various dimensions. A simplified version of the algorithm has also been used to find the saddle points of symmetric systems with the help of some analytical knowledge. The main advantages and effectiveness of the method are clearly illustrated with some examples. Promising results of our method are shown on various problems with varied degrees of freedom.

ELITE: ensemble of optimal input-pruned neural networks using TRUST-TECH.

The ensemble of optimal input-pruned neural networks using TRUST-TECH (ELITE) method for constructing high-quality ensemble through an optimal linear combination of accurate and diverse neural networks is developed. The optimization problems in the proposed methodology are solved by a global optimization a global optimization method called TRansformation Under Stability-reTraining Equilibrium Characterization (TRUST-TECH), whose main features include its capability in identifying multiple local optimal solutions in a deterministic, systematic, and tier-by-tier manner. ELITE creates a diverse population via a feature selection procedure of different local optimal neural networks obtained using tier-1 TRUST-TECH search. In addition, the capability of each input-pruned network is fully exploited through a TRUST-TECH-based optimal training. Finally, finding the optimal linear combination weights for an ensemble is modeled as a nonlinear programming problem and solved using TRUST-TECH and the interior point method, where the issue of non-convexity can be effectively handled. Extensive numerical experiments have been carried out for pattern classification on the synthetic and benchmark datasets. Numerical results show that ELITE consistently outperforms existing methods on the benchmark datasets. The results show that ELITE can be very promising for constructing high-quality neural network ensembles.

Refining motifs by improving information content scores using neighborhood profile search.

The main goal of the motif finding problem is to detect novel, over-represented unknown signals in a set of sequences (e.g. transcription factor binding sites in a genome). The most widely used algorithms for finding motifs obtain a generative probabilistic representation of these over-represented signals and try to discover profiles that maximize the information content score. Although these profiles form a very powerful representation of the signals, the major difficulty arises from the fact that the best motif corresponds to the global maximum of a non-convex continuous function. Popular algorithms like Expectation Maximization (EM) and Gibbs sampling tend to be very sensitive to the initial guesses and are known to converge to the nearest local maximum very quickly. In order to improve the quality of the results, EM is used with multiple random starts or any other powerful stochastic global methods that might yield promising initial guesses (like projection algorithms). Global methods do not necessarily give initial guesses in the convergence region of the best local maximum but rather suggest that a promising solution is in the neighborhood region. In this paper, we introduce a novel optimization framework that searches the neighborhood regions of the initial alignment in a systematic manner to explore the multiple local optimal solutions. This effective search is achieved by transforming the original optimization problem into its corresponding dynamical system and estimating the practical stability boundary of the local maximum. Our results show that the popularly used EM algorithm often converges to sub-optimal solutions which can be significantly improved by the proposed neighborhood profile search. Based on experiments using both synthetic and real datasets, our method demonstrates significant improvements in the information content scores of the probabilistic models. The proposed method also gives the flexibility in using different local solvers and global methods depending on their suitability for some specific datasets.