Large-Scale Multiobjective Evolutionary Algorithm Guided by Low-Dimensional Surrogates of Scalarization Functions.

Recently, computationally intensive multiobjective optimization problems have been efficiently solved by surrogate-assisted multiobjective evolutionary algorithms. However, most of those algorithms can only handle no more than 200 decision variables. As the number of decision variables increases further, unreliable surrogate models will result in a dramatic deterioration of their performance, which makes large-scale expensive multiobjective optimization challenging. To address this challenge, we develop a large-scale multiobjective evolutionary algorithm guided by low-dimensional surrogate models of scalarization functions. The proposed algorithm (termed LDS-AF) reduces the dimension of the original decision space based on principal component analysis, and then directly approximates the scalarization functions in a decompositionbased multiobjective evolutionary algorithm. With the help of a two-stage modeling strategy and convergence control strategy, LDS-AF can keep a good balance between convergence and diversity, and achieve a promising performance without being trapped in a local optimum prematurely. The experimental results on a set of test instances have demonstrated its superiority over eight state-of-the-art algorithms on multiobjective optimization problems with up to 1000 decision variables using only 500 real function evaluations.

IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing.

Image anomaly detection (IAD) is an emerging and vital computer vision task in industrial manufacturing (IM). Recently, many advanced algorithms have been reported, but their performance deviates considerably with various IM settings. We realize that the lack of a uniform IM benchmark is hindering the development and usage of IAD methods in real-world applications. In addition, it is difficult for researchers to analyze IAD algorithms without a uniform benchmark. To solve this problem, we propose a uniform IM benchmark, for the first time, to assess how well these algorithms perform, which includes various levels of supervision (unsupervised versus fully supervised), learning paradigms (few-shot, continual and noisy label), and efficiency (memory usage and inference speed). Then, we construct a comprehensive IAD benchmark (IM-IAD), which includes 19 algorithms on seven major datasets with a uniform setting. Extensive experiments (17 017 total) on IM-IAD provide in-depth insights into IAD algorithm redesign or selection. Moreover, the proposed IM-IAD benchmark challenges existing algorithms and suggests future research directions. For reproducibility and accessibility, the source code is uploaded to the website: https://github.com/M-3LAB/open-iad.

Learning to Guide Particle Search for Dynamic Multiobjective Optimization.

Dynamic multiobjective optimization problems (DMOPs) are characterized by multiple objectives that change over time in varying environments. More specifically, environmental changes can be described as various dynamics. However, it is difficult for existing dynamic multiobjective algorithms (DMOAs) to handle DMOPs due to their inability to learn in different environments to guide the search. Besides, solving DMOPs is typically an online task, requiring low computational cost of a DMOA. To address the above challenges, we propose a particle search guidance network (PSGN), capable of directing individuals' search actions, including learning target selection and acceleration coefficient control. PSGN can learn the actions that should be taken in each environment through rewarding or punishing the network by reinforcement learning. Thus, PSGN is capable of tackling DMOPs of various dynamics. Additionally, we efficiently adjust PSGN hidden nodes and update the output weights in an incremental learning way, enabling PSGN to direct particle search at a low computational cost. We compare the proposed PSGN with seven state-of-the-art algorithms, and the excellent performance of PSGN verifies that it can handle DMOPs of various dynamics in a computationally very efficient way.

DRL-Searcher: A Unified Approach to Multirobot Efficient Search for a Moving Target.

This article studies the multirobot efficient search (MuRES) for a nonadversarial moving target problem, whose objective is usually defined as either minimizing the target's expected capture time or maximizing the target's capture probability within a given time budget. Different from canonical MuRES algorithms, which target only one specific objective, our proposed algorithm, named distributional reinforcement learning-based searcher (DRL-Searcher), serves as a unified solution to both MuRES objectives. DRL-Searcher employs distributional reinforcement learning (DRL) to evaluate the full distribution of a given search policy's return, that is, the target's capture time, and thereafter makes improvements with respect to the particularly specified objective. We further adapt DRL-Searcher to the use case without the target's real-time location information, where only the probabilistic target belief (PTB) information is provided. Lastly, the recency reward is designed for implicit coordination among multiple robots. Comparative simulation results in a range of MuRES test environments show the superior performance of DRL-Searcher to state of the arts. Additionally, we deploy DRL-Searcher to a real multirobot system for moving target search in a self-constructed indoor environment with satisfying results.

Efficient Surrogate Modeling Method for Evolutionary Algorithm to Solve Bilevel Optimization Problems.

The purpose of this study was to develop an evolutionary algorithm (EA) with bilevel surrogate modeling, called BL-SAEA, for tackling bilevel optimization problems (BLOPs), in which an upper level problem is to be solved subject to the optimality of a corresponding lower level problem. The motivation of this article is that the extensive lower level optimization required by each upper level solution consumes too many function evaluations, leading to poor optimization performance of EAs. To this end, during the upper level optimization, the BL-SAEA builds an upper level surrogate model to select several promising upper level solutions for the lower level optimization. Because only a small number of upper level solutions require the lower level optimization, the number of function evaluations can be considerably reduced. During the lower level optimization, the BL-SAEA constructs multiple lower level surrogate models to initialize the population of the lower level optimization, thus further decreasing the number of function evaluations. Experimental results on two widely used benchmarks and two real-world BLOPs demonstrate the superiority of our proposed algorithm over six state-of-the-art algorithms in terms of effectiveness and efficiency.

A Generic Self-Supervised Framework of Learning Invariant Discriminative Features.

Self-supervised learning (SSL) has become a popular method for generating invariant representations without the need for human annotations. Nonetheless, the desired invariant representation is achieved by utilizing prior online transformation functions on the input data. As a result, each SSL framework is customized for a particular data type, for example, visual data, and further modifications are required if it is used for other dataset types. On the other hand, autoencoder (AE), which is a generic and widely applicable framework, mainly focuses on dimension reduction and is not suited for learning invariant representation. This article proposes a generic SSL framework based on a constrained self-labeling assignment process that prevents degenerate solutions. Specifically, the prior transformation functions are replaced with a self-transformation mechanism, derived through an unsupervised training process of adversarial training, for imposing invariant representations. Via the self-transformation mechanism, pairs of augmented instances can be generated from the same input data. Finally, a training objective based on contrastive learning is designed by leveraging both the self-labeling assignment and the self-transformation mechanism. Despite the fact that the self-transformation process is very generic, the proposed training strategy outperforms a majority of state-of-the-art representation learning methods based on AE structures. To validate the performance of our method, we conduct experiments on four types of data, namely visual, audio, text, and mass spectrometry data and compare them in terms of four quantitative metrics. Our comparison results demonstrate that the proposed method is effective and robust in identifying patterns within the tested datasets.

A Data Stream Ensemble Assisted Multifactorial Evolutionary Algorithm for Offline Data-Driven Dynamic Optimization.

Existing work on offline data-driven optimization mainly focuses on problems in static environments, and little attention has been paid to problems in dynamic environments. Offline data-driven optimization in dynamic environments is a challenging problem because the distribution of collected data varies over time, requiring surrogate models and optimal solutions tracking with time. This paper proposes a knowledge-transfer-based data-driven optimization algorithm to address these issues. First, an ensemble learning method is adopted to train surrogate models to leverage the knowledge of data in historical environments as well as adapt to new environments. Specifically, given data in a new environment, a model is constructed with the new data, and the preserved models of historical environments are further trained with the new data. Then, these models are considered to be base learners and combined as an ensemble surrogate model. After that, all base learners and the ensemble surrogate model are simultaneously optimized in a multitask environment for finding optimal solutions for real fitness functions. In this way, the optimization tasks in the previous environments can be used to accelerate the tracking of the optimum in the current environment. Since the ensemble model is the most accurate surrogate, we assign more individuals to the ensemble surrogate than its base learners. Empirical results on six dynamic optimization benchmark problems demonstrate the effectiveness of the proposed algorithm compared with four state-of-the-art offline data-driven optimization algorithms. Code is available at https://github.com/Peacefulyang/DSE_MFS.git.

Performance Indicator-Based Adaptive Model Selection for Offline Data-Driven Multiobjective Evolutionary Optimization.

A number of real-world multiobjective optimization problems (MOPs) are driven by the data from experiments or computational simulations. In some cases, no new data can be sampled during the optimization process and only a certain amount of data can be sampled before optimization starts. Such problems are known as offline data-driven MOPs. Although multiple surrogate models approximating each objective function are able to replace the real fitness evaluations in evolutionary algorithms (EAs), their approximation errors are easily accumulated and therefore, mislead the solution ranking. To mitigate this issue, a new surrogate-assisted indicator-based EA for solving offline data-driven multiobjective problems is proposed. The proposed algorithm adopts an indicator-based selection EA as the baseline optimizer due to its selection robustness to the approximation errors of surrogate models. Both the Kriging models and radial basis function networks (RBFNs) are employed as surrogate models. An adaptive model selection mechanism is designed to choose the right type of models according to a maximum acceptable approximation error that is less likely to mislead the indicator-based search. The main idea is that when the uncertainty of the Kriging models exceeds the acceptable error, the proposed algorithm selects RBFNs as the surrogate models. The results comparing with state-of-the-art algorithms on benchmark problems with up to ten objectives indicate that the proposed algorithm is effective on offline data-driven optimization problems with up to 20 and 30 decision variables.

Evolutionary Large-Scale Dynamic Optimization Using Bilevel Variable Grouping.

Variable grouping provides an efficient approach to large-scale optimization, and multipopulation strategies are effective for both large-scale optimization and dynamic optimization. However, variable grouping is not well studied in large-scale dynamic optimization when cooperating with multipopulation strategies. Specifically, when the numbers/sizes of the variable subcomponents are large, the performance of the algorithms will be substantially degraded. To address this issue, we propose a bilevel variable grouping (BLVG)-based framework. First, the primary grouping applies a state-of-the-art variable grouping method based on variable interaction analysis to group the variables into subcomponents. Second, the secondary grouping further groups the subcomponents into variable cells, that is, combination variable cells and decomposition variable cells. We then tailor a multipopulation strategy to process the two types of variable cells efficiently in a cooperative coevolutionary (CC) way. As indicated by the empirical study on large-scale dynamic optimization problems (DOPs) of up to 300 dimensions, the proposed framework outperforms several state-of-the-art frameworks for large-scale dynamic optimization.

Solution Set Augmentation for Knee Identification in Multiobjective Decision Analysis.

In multiobjective decision making, most knee identification algorithms implicitly assume that the given solutions are well distributed and can provide sufficient information for identifying knee solutions. However, this assumption may fail to hold when the number of objectives is large or when the shape of the Pareto front is complex. To address the above issues, we propose a knee-oriented solution augmentation (KSA) framework that converts the Pareto front into a multimodal auxiliary function whose basins correspond to the knee regions of the Pareto front. The auxiliary function is then approximated using a surrogate and its basins are identified by a peak detection method. Additional solutions are then generated in the detected basins in the objective space and mapped to the decision space with the help of an inverse model. These solutions are evaluated by the original objective functions and added to the given solution set. To assess the quality of the augmented solution set, a measurement is proposed for the verification of knee solutions when the true Pareto front is unknown. The effectiveness of KSA is verified on widely used benchmark problems and successfully applied to a hybrid electric vehicle controller design problem.

Adaptive Prototypical Networks With Label Words and Joint Representation Learning for Few-Shot Relation Classification.

Relation classification (RC) task is one of fundamental tasks of information extraction, aiming to detect the relation information between entity pairs in unstructured natural language text and generate structured data in the form of entity-relation triple. Although distant supervision methods can effectively alleviate the problem of lack of training data in supervised learning, they also introduce noise into the data and still cannot fundamentally solve the long-tail distribution problem of the training instances. In order to enable the neural network to learn new knowledge through few instances such as humans, this work focuses on few-shot relation classification (FSRC), where a classifier should generalize to new classes that have not been seen in the training set, given only a number of samples for each class. To make full use of the existing information and get a better feature representation for each instance, we propose to encode each class prototype in an adaptive way from two aspects. First, based on the prototypical networks, we propose an adaptive mixture mechanism to add label words to the representation of the class prototype, which, to the best of our knowledge, is the first attempt to integrate the label information into features of the support samples of each class so as to get more interactive class prototypes. Second, to more reasonably measure the distances between samples of each category, we introduce a loss function for joint representation learning (JRL) to encode each support instance in an adaptive manner. Extensive experiments have been conducted on FewRel under different few-shot (FS) settings, and the results show that the proposed adaptive prototypical networks with label words and JRL has not only achieved significant improvements in accuracy but also increased the generalization ability of FSRC.

Evolving Dual-Threshold Bienenstock-Cooper-Munro Learning Rules in Echo State Networks.

The strengthening and the weakening of synaptic strength in existing Bienenstock-Cooper-Munro (BCM) learning rule are determined by a long-term potentiation (LTP) sliding modification threshold and the afferent synaptic activities. However, synaptic long-term depression (LTD) even affects low-active synapses during the induction of synaptic plasticity, which may lead to information loss. Biological experiments have found another LTD threshold that can induce either potentiation or depression or no change, even at the activated synapses. In addition, existing BCM learning rules can only select a set of fixed rule parameters, which is biologically implausible and practically inflexible to learn the structural information of input signals. In this article, an evolved dual-threshold BCM learning rule is proposed to regulate the reservoir internal connection weights of the echo-state-network (ESN), which can contribute to alleviating information loss and enhancing learning performance by introducing different optimal LTD thresholds for different postsynaptic neurons. Our experimental results show that the evolved dual-threshold BCM learning rule can result in the synergistic learning of different plasticity rules, effectively improving the learning performance of an ESN in comparison with existing neural plasticity learning rules and some state-of-the-art ESN variants on three widely used benchmark tasks and the prediction of an esterification process.

An Adaptive Reference Vector-Guided Evolutionary Algorithm Using Growing Neural Gas for Many-Objective Optimization of Irregular Problems.

Most reference vector-based decomposition algorithms for solving multiobjective optimization problems may not be well suited for solving problems with irregular Pareto fronts (PFs) because the distribution of predefined reference vectors may not match well with the distribution of the Pareto-optimal solutions. Thus, the adaptation of the reference vectors is an intuitive way for decomposition-based algorithms to deal with irregular PFs. However, most existing methods frequently change the reference vectors based on the activeness of the reference vectors within specific generations, slowing down the convergence of the search process. To address this issue, we propose a new method to learn the distribution of the reference vectors using the growing neural gas (GNG) network to achieve automatic yet stable adaptation. To this end, an improved GNG is designed for learning the topology of the PFs with the solutions generated during a period of the search process as the training data. We use the individuals in the current population as well as those in previous generations to train the GNG to strike a balance between exploration and exploitation. Comparative studies conducted on popular benchmark problems and a real-world hybrid vehicle controller design problem with complex and irregular PFs show that the proposed method is very competitive.

Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis.

Skin lesion diagnosis is a key step for skin cancer screening, which requires high accuracy and interpretability. Though many computer-aided methods, especially deep learning methods, have made remarkable achievements in skin lesion diagnosis, their generalization and interpretability are still a challenge. To solve this issue, we propose an interpretability-based multimodal convolutional neural network (IM-CNN), which is a multiclass classification model with skin lesion images and metadata of patients as input for skin lesion diagnosis. The structure of IM-CNN consists of three main paths to deal with metadata, features extracted from segmented skin lesion with domain knowledge, and skin lesion images, respectively. We add interpretable visual modules to provide explanations for both images and metadata. In addition to area under the ROC curve (AUC), sensitivity, and specificity, we introduce a new indicator, an AUC curve with a sensitivity larger than 80% (AUC_SEN_80) for performance evaluation. Extensive experimental studies are conducted on the popular HAM10000 dataset, and the results indicate that the proposed model has overwhelming advantages compared with popular deep learning models, such as DenseNet, ResNet, and other state-of-the-art models for melanoma diagnosis. The proposed multimodal model also achieves on average 72% and 21% improvement in terms of sensitivity and AUC_SEN_80, respectively, compared with the single-modal model. The visual explanations can also help gain trust from dermatologists and realize man-machine collaborations, effectively reducing the limitation of black-box models in supporting medical decision making.

Ternary Compression for Communication-Efficient Federated Learning.

Learning over massive data stored in different locations is essential in many real-world applications. However, sharing data is full of challenges due to the increasing demands of privacy and security with the growing use of smart mobile devices and Internet of thing (IoT) devices. Federated learning provides a potential solution to privacy-preserving and secure machine learning, by means of jointly training a global model without uploading data distributed on multiple devices to a central server. However, most existing work on federated learning adopts machine learning models with full-precision weights, and almost all these models contain a large number of redundant parameters that do not need to be transmitted to the server, consuming an excessive amount of communication costs. To address this issue, we propose a federated trained ternary quantization (FTTQ) algorithm, which optimizes the quantized networks on the clients through a self-learning quantization factor. Theoretical proofs of the convergence of quantization factors, unbiasedness of FTTQ, as well as a reduced weight divergence are given. On the basis of FTTQ, we propose a ternary federated averaging protocol (T-FedAvg) to reduce the upstream and downstream communication of federated learning systems. Empirical experiments are conducted to train widely used deep learning models on publicly available data sets, and our results demonstrate that the proposed T-FedAvg is effective in reducing communication costs and can even achieve slightly better performance on non-IID data in contrast to the canonical federated learning algorithms.

A Pattern Mining-Based Evolutionary Algorithm for Large-Scale Sparse Multiobjective Optimization Problems.

In real-world applications, there exist a lot of multiobjective optimization problems whose Pareto-optimal solutions are sparse, that is, most variables of these solutions are 0. Generally, many sparse multiobjective optimization problems (SMOPs) contain a large number of variables, which pose grand challenges for evolutionary algorithms to find the optimal solutions efficiently. To address the curse of dimensionality, this article proposes an evolutionary algorithm for solving large-scale SMOPs, which aims to mine the sparse distribution of the Pareto-optimal solutions and, thus, considerably reduces the search space. More specifically, the proposed algorithm suggests an evolutionary pattern mining approach to detect the maximum and minimum candidate sets of the nonzero variables in the Pareto-optimal solutions, and uses them to limit the dimensions in generating offspring solutions. For further performance enhancement, a binary crossover operator and a binary mutation operator are designed to ensure the sparsity of solutions. According to the results on eight benchmark problems and four real-world problems, the proposed algorithm is superior over existing evolutionary algorithms in solving large-scale SMOPs.

Transfer Learning Based Co-Surrogate Assisted Evolutionary Bi-Objective Optimization for Objectives with Non-Uniform Evaluation Times.

Most existing multiobjective evolutionary algorithms (MOEAs) implicitly assume that each objective function can be evaluated within the same period of time. Typically. this is untenable in many real-world optimization scenarios where evaluation of different objectives involves different computer simulations or physical experiments with distinct time complexity. To address this issue, a transfer learning scheme based on surrogate-assisted evolutionary algorithms (SAEAs) is proposed, in which a co-surrogate is adopted to model the functional relationship between the fast and slow objective functions and a transferable instance selection method is introduced to acquire useful knowledge from the search process of the fast objective. Our experimental results on DTLZ and UF test suites demonstrate that the proposed algorithm is competitive for solving bi-objective optimization where objectives have non-uniform evaluation times.

Image Clustering Using an Augmented Generative Adversarial Network and Information Maximization.

Image clustering has recently attracted significant attention due to the increased availability of unlabeled datasets. The efficiency of traditional clustering algorithms heavily depends on the distance functions used and the dimensionality of the features. Therefore, performance degradation is often observed when tackling either unprocessed images or high-dimensional features extracted from processed images. To deal with these challenges, we propose a deep clustering framework consisting of a modified generative adversarial network (GAN) and an auxiliary classifier. The modification employs Sobel operations prior to the discriminator of the GAN to enhance the separability of the learned features. The discriminator is then leveraged to generate representations as to the input to an auxiliary classifier. An objective function is utilized to train the auxiliary classifier by maximizing the mutual information between the representations obtained via the discriminator model and the same representations perturbed via adversarial training. We further improve the robustness of the auxiliary classifier by introducing a penalty term into the objective function. This minimizes the divergence across multiple transformed representations generated by the discriminator model with a low dropout rate. The auxiliary classifier is implemented with a group of multiple cluster-heads, where a tolerance hyper-parameter is used to tackle imbalanced data. Our results indicate that the proposed method achieves competitive results compared with state-of-the-art clustering methods on a wide range of benchmark datasets including CIFAR-10, CIFAR-100/20, and STL10.

Balancing Objective Optimization and Constraint Satisfaction in Constrained Evolutionary Multiobjective Optimization.

Both objective optimization and constraint satisfaction are crucial for solving constrained multiobjective optimization problems, but the existing evolutionary algorithms encounter difficulties in striking a good balance between them when tackling complex feasible regions. To address this issue, this article proposes a two-stage evolutionary algorithm, which adjusts the fitness evaluation strategies during the evolutionary process to adaptively balance objective optimization and constraint satisfaction. The proposed algorithm can switch between the two stages according to the status of the current population, enabling the population to cross the infeasible region and reach the feasible regions in one stage, and to spread along the feasible boundaries in the other stage. Experimental studies on four benchmark suites and three real-world applications demonstrate the superiority of the proposed algorithm over the state-of-the-art algorithms, especially on problems with complex feasible regions.

A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks.

It has been widely recognized that the efficient training of neural networks (NNs) is crucial to classification performance. While a series of gradient-based approaches have been extensively developed, they are criticized for the ease of trapping into local optima and sensitivity to hyperparameters. Due to the high robustness and wide applicability, evolutionary algorithms (EAs) have been regarded as a promising alternative for training NNs in recent years. However, EAs suffer from the curse of dimensionality and are inefficient in training deep NNs (DNNs). By inheriting the advantages of both the gradient-based approaches and EAs, this article proposes a gradient-guided evolutionary approach to train DNNs. The proposed approach suggests a novel genetic operator to optimize the weights in the search space, where the search direction is determined by the gradient of weights. Moreover, the network sparsity is considered in the proposed approach, which highly reduces the network complexity and alleviates overfitting. Experimental results on single-layer NNs, deep-layer NNs, recurrent NNs, and convolutional NNs (CNNs) demonstrate the effectiveness of the proposed approach. In short, this work not only introduces a novel approach for training DNNs but also enhances the performance of EAs in solving large-scale optimization problems.