Granger causality using Jacobian in neural networks.

Granger causality is a commonly used method for uncovering information flow and dependencies in a time series. Here, we introduce JGC (Jacobian Granger causality), a neural network-based approach to Granger causality using the Jacobian as a measure of variable importance, and propose a variable selection procedure for inferring Granger causal variables with this measure, using criteria of significance and consistency. The resulting approach performs consistently well compared to other approaches in identifying Granger causal variables, the associated time lags, as well as interaction signs. In addition, we also discuss the need for contemporaneous variables in Granger causal modeling as well as how these neural network-based approaches reduce the impact of nonseparability in dynamical systems, a problem where predictive information on a target variable is not unique to its causes, but also contained in the history of the target variable itself.

Temperature dependent structural variations of OH(-)(H2O)n, n = 4-7: effects on vibrational and photoelectron spectra.

In this work, we identified a large number of structurally distinct isomers of midsized deprotonated water clusters, OH(-)(H2O)n=4-7, using first-principles methods. The temperature dependence of the structural variation in the solvation shell of OH(-) for these clusters was examined under the harmonic superposition approximation. We simulated the vibrational and photoelectron spectra based on these thermodynamic calculations. We found that the isomers with 3-coordinated hydroxide dominate the population in these midsized clusters. Furthermore, an increase in temperature causes a topological change from compact isomers with many intermolecular hydrogen bonds to open isomers with fewer but more directional intermolecular hydrogen bonds. We showed that this evolution in structure can be observed through the change in the vibrational spectra at 3200-3400 cm(-1). In addition, the increase in directional hydrogen bonded isomers, which have outer hydration shell with OH bonds pointing to the hydroxide, causes the vertical detachment energy to increase at higher temperatures. Lastly, we also performed studies to understand the variation in the aforementioned spectral quantities with the variation in the coordination number of the hydroxide.

Generalizable and Discriminative Representations for Adversarially Robust Few-Shot Learning.

Few-shot image classification (FSIC) is beneficial for a variety of real-world scenarios, aiming to construct a recognition system with limited training data. In this article, we extend the original FSIC task by incorporating defense against malicious adversarial examples. This can be an arduous challenge because numerous deep learning-based approaches remain susceptible to adversarial examples, even when trained with ample amounts of data. Previous studies on this problem have predominantly concentrated on the meta-learning framework, which involves sampling numerous few-shot tasks during the training stage. In contrast, we propose a straightforward but effective baseline via learning robust and discriminative representations without tedious meta-task sampling, which can further be generalized to unforeseen adversarial FSIC tasks. Specifically, we introduce an adversarial-aware (AA) mechanism that exploits feature-level distinctions between the legitimate and the adversarial domains to provide supplementary supervision. Moreover, we design a novel adversarial reweighting training strategy to ameliorate the imbalance among adversarial examples. To further enhance the adversarial robustness without compromising discriminative features, we propose the cyclic feature purifier during the postprocessing projection, which can reduce the interference of unforeseen adversarial examples. Furthermore, our method can obtain robust feature embeddings that maintain superior transferability, even when facing cross-domain adversarial examples. Extensive experiments and systematic analyses demonstrate that our method achieves state-of-the-art robustness as well as natural performance among adversarially robust FSIC algorithms on three standard benchmarks by a substantial margin.

Chance-Constrained Multiple-Choice Knapsack Problem: Model, Algorithms, and Applications.

The multiple-choice knapsack problem (MCKP) is a classic NP-hard combinatorial optimization problem. Motivated by several significant real-world applications, this work investigates a novel variant of MCKP called the chance-constrained MCKP (CCMCKP), where item weights are random variables. In particular, we focus on the practical scenario of CCMCKP, in which the probability distributions of random weights are unknown and only sample data is available. We first present the problem formulation of CCMCKP and then establish the two benchmark sets. The first set contains synthetic instances, while the second set is designed to simulate a real-world application scenario of a telecommunication company. To solve CCMCKP, we propose a data-driven adaptive local search (DDALS) algorithm. Compared to existing stochastic optimization and distributionally robust optimization methods, the main novelty of DDALS lies in its data-driven solution evaluation approach, which does not make any assumptions about the underlying distributions and is highly effective even when faced with a high intensity of the chance constraint and a limited amount of sample data. Experimental results demonstrate the superiority of DDALS over the baselines on both the benchmarks. Finally, DDALS can serve as the baseline for future research, and the benchmark sets are open-sourced to further promote research on this challenging problem.

Evolution by adapting surrogates.

To deal with complex optimization problems plagued with computationally expensive fitness functions, the use of surrogates to replace the original functions within the evolutionary framework is becoming a common practice. However, the appropriate datacentric approximation methodology to use for the construction of surrogate model would depend largely on the nature of the problem of interest, which varies from fitness landscape and state of the evolutionary search, to the characteristics of search algorithm used. This has given rise to the plethora of surrogate-assisted evolutionary frameworks proposed in the literature with ad hoc approximation/surrogate modeling methodologies considered. Since prior knowledge on the suitability of the data centric approximation methodology to use in surrogate-assisted evolutionary optimization is typically unavailable beforehand, this paper presents a novel evolutionary framework with the evolvability learning of surrogates (EvoLS) operating on multiple diverse approximation methodologies in the search. Further, in contrast to the common use of fitness prediction error as a criterion for the selection of surrogates, the concept of evolvability to indicate the productivity or suitability of an approximation methodology that brings about fitness improvement in the evolutionary search is introduced as the basis for adaptation. The backbone of the proposed EvoLS is a statistical learning scheme to determine the evolvability of each approximation methodology while the search progresses online. For each individual solution, the most productive approximation methodology is inferred, that is, the method with highest evolvability measure. Fitness improving surrogates are subsequently constructed for use within a trust-region enabled local search strategy, leading to the self-configuration of a surrogate-assisted memetic algorithm for solving computationally expensive problems. A numerical study of EvoLS on commonly used benchmark problems and a real-world computationally expensive aerodynamic car rear design problem highlights the efficacy of the proposed EvoLS in attaining reliable, high quality, and efficient performance under a limited computational budget.

Scalable Transfer Evolutionary Optimization: Coping With Big Task Instances.

In today's digital world, we are faced with an explosion of data and models produced and manipulated by numerous large-scale cloud-based applications. Under such settings, existing transfer evolutionary optimization (TrEO) frameworks grapple with simultaneously satisfying two important quality attributes, namely: 1) scalability against a growing number of source tasks and 2) online learning agility against sparsity of relevant sources to the target task of interest. Satisfying these attributes shall facilitate practical deployment of transfer optimization to scenarios with big task instances, while curbing the threat of negative transfer. While applications of existing algorithms are limited to tens of source tasks, in this article, we take a quantum leap forward in enabling more than two orders of magnitude scale-up in the number of tasks; that is, we efficiently handle scenarios beyond 1000 source task instances. We devise a novel TrEO framework comprising two co-evolving species for joint evolutions in the space of source knowledge and in the search space of solutions to the target problem. In particular, co-evolution enables the learned knowledge to be orchestrated on the fly, expediting convergence in the target optimization task. We have conducted an extensive series of experiments across a set of practically motivated discrete and continuous optimization examples comprising a large number of source task instances, of which only a small fraction indicate source-target relatedness. The experimental results show that not only does our proposed framework scale efficiently with a growing number of source tasks but is also effective in capturing relevant knowledge against sparsity of related sources, fulfilling the two salient features of scalability and online learning agility.

Adversary Agnostic Robust Deep Reinforcement Learning.

Deep reinforcement learning (DRL) policies have been shown to be deceived by perturbations (e.g., random noise or intensional adversarial attacks) on state observations that appear at test time but are unknown during training. To increase the robustness of DRL policies, previous approaches assume that explicit adversarial information can be added into the training process, to achieve generalization ability on these perturbed observations as well. However, such approaches not only make robustness improvement more expensive but may also leave a model prone to other kinds of attacks in the wild. In contrast, we propose an adversary agnostic robust DRL paradigm that does not require learning from predefined adversaries. To this end, we first theoretically show that robustness could indeed be achieved independently of the adversaries based on a policy distillation (PD) setting. Motivated by this finding, we propose a new PD loss with two terms: 1) a prescription gap maximization (PGM) loss aiming to simultaneously maximize the likelihood of the action selected by the teacher policy and the entropy over the remaining actions and 2) a corresponding Jacobian regularization (JR) loss that minimizes the magnitude of gradients with respect to the input state. The theoretical analysis substantiates that our distillation loss guarantees to increase the prescription gap and hence improves the adversarial robustness. Furthermore, experiments on five Atari games firmly verify the superiority of our approach compared to the state-of-the-art baselines.

GrpAvoid: Multigroup Collision-Avoidance Control and Optimization for UAV Swarm.

Collision-avoidance control for UAV swarm has recently drawn great attention due to its significant implications in many industrial and commercial applications. However, traditional collision-avoidance models for UAV swarm tend to focus on avoidance at individual UAV level, and no explicit strategy is designed for avoidance among multiple UAV groups. When directly applying these models for multigroup UAV scenarios, the deadlock situation may happen. A group of UAVs may be temporally blocked by other groups in a narrow space and cannot progress toward achieving its goal. To this end, this article proposes a modeling and optimization approach to multigroup UAV collision avoidance. Specifically, group level collision detection and adaption mechanism are introduced, efficiently detecting potential collisions among different UAV groups and restructuring a group into subgroups for better collision and deadlock avoidance. A two-level control model is then designed for realizing collision avoidance among UAV groups and of UAVs within each group. Finally, an evolutionary multitask optimization method is introduced to effectively calibrate the parameters that exist in different levels of our control model, and an adaptive fitness evaluation strategy is proposed to reduce computation overhead in simulation-based optimization. The simulation results show that our model has superior performances in deadlock resolution, motion stability, and distance maintenance in multigroup UAV scenarios compared to the state-of-the-art collision-avoidance models. The model optimization results also show that our model optimization method can largely reduce execution time for computationally-intensive optimization process that involves UAV swarm simulation.

Adaptive Transfer Kernel Learning for Transfer Gaussian Process Regression.

Transfer regression is a practical and challenging problem with important applications in various domains, such as engineering design and localization. Capturing the relatedness of different domains is the key of adaptive knowledge transfer. In this paper, we investigate an effective way of explicitly modelling domain relatedness through transfer kernel, a transfer-specified kernel that considers domain information in the covariance calculation. Specifically, we first give the formal definition of transfer kernel, and introduce three basic general forms that well cover existing related works. To cope with the limitations of the basic forms in handling complex real-world data, we further propose two advanced forms. Corresponding instantiations of the two forms are developed, namely Trkαß and Trkω based on multiple kernel learning and neural networks, respectively. For each instantiation, we present a condition with which the positive semi-definiteness is guaranteed and a semantic meaning is interpreted to the learned domain relatedness. Moreover, the condition can be easily used in the learning of TrGP αß and TrGP ω that are the Gaussian process models with the transfer kernels Trkαß and Trkω respectively. Extensive empirical studies show the effectiveness of TrGP αß and TrGP ω on domain relatedness modelling and transfer adaptiveness.

Transfer Kernel Learning for Multi-Source Transfer Gaussian Process Regression.

Multi-source transfer regression is a practical and challenging problem where capturing the diverse relatedness of different domains is the key of adaptive knowledge transfer. In this article, we propose an effective way of explicitly modeling the domain relatedness of each domain pair through transfer kernel learning. Specifically, we first discuss the advantages and disadvantages of existing transfer kernels in handling the multi-source transfer regression problem. To cope with the limitations of the existing transfer kernels, we further propose a novel multi-source transfer kernel kms. The proposed kms assigns a learnable parametric coefficient to model the relatedness of each inter-domain pair, and simultaneously regulates the relatedness of the intra-domain pair to be 1. Moreover, to capture the heterogeneous data characteristics of multiple domains, kms exploits different standard kernels for different domain pairs. We further provide a theorem that not only guarantees the positive semi-definiteness of kms but also conveys a semantic interpretation to the learned domain relatedness. Moreover, the theorem can be easily used in the learning of the corresponding transfer Gaussian process model with kms. Extensive empirical studies show the effectiveness of our proposed method on domain relatedness modelling and transfer performance.

Semisupervised Graph Neural Networks for Graph Classification.

Graph classification aims to predict the label associated with a graph and is an important graph analytic task with widespread applications. Recently, graph neural networks (GNNs) have achieved state-of-the-art results on purely supervised graph classification by virtue of the powerful representation ability of neural networks. However, almost all of them ignore the fact that graph classification usually lacks reasonably sufficient labeled data in practical scenarios due to the inherent labeling difficulty caused by the high complexity of graph data. The existing semisupervised GNNs typically focus on the task of node classification and are incapable to deal with graph classification. To tackle the challenging but practically useful scenario, we propose a novel and general semisupervised GNN framework for graph classification, which takes full advantage of a slight amount of labeled graphs and abundant unlabeled graph data. In our framework, we train two GNNs as complementary views for collaboratively learning high-quality classifiers using both labeled and unlabeled graphs. To further exploit the view itself, we constantly select pseudo-labeled graph examples with high confidence from its own view for enlarging the labeled graph dataset and enhancing predictions on graphs. Furthermore, the proposed framework is investigated on two specific implementation regimes with a few labeled graphs and the extremely few labeled graphs, respectively. Extensive experimental results demonstrate the effectiveness of our proposed semisupervised GNN framework for graph classification on several benchmark datasets.

Generative Multiform Bayesian Optimization.

Many real-world problems, such as airfoil design, involve optimizing a black-box expensive objective function over complex-structured input space (e.g., discrete space or non-Euclidean space). By mapping the complex-structured input space into a latent space of dozens of variables, a two-stage procedure labeled as generative model-based optimization (GMO), in this article, shows promise in solving such problems. However, the latent dimension of GMO is hard to determine, which may trigger the conflicting issue between desirable solution accuracy and convergence rate. To address the above issue, we propose a multiform GMO approach, namely, generative multiform optimization (GMFoO), which conducts optimization over multiple latent spaces simultaneously to complement each other. More specifically, we devise a generative model which promotes a positive correlation between latent spaces to facilitate effective knowledge transfer in GMFoO. And furthermore, by using Bayesian optimization (BO) as the optimizer, we propose two strategies to exchange information between these latent spaces continuously. Experimental results are presented on airfoil and corbel design problems and an area maximization problem as well to demonstrate that our proposed GMFoO converges to better designs on a limited computational budget.

A Cell-Based Fast Memetic Algorithm for Automated Convolutional Neural Architecture Design.

Neural architecture search (NAS) has attracted much attention in recent years. It automates the neural network construction for different tasks, which is traditionally addressed manually. In the literature, evolutionary optimization (EO) has been proposed for NAS due to its strong global search capability. However, despite the success enjoyed by EO, it is worth noting that existing EO algorithms for NAS are often very computationally expensive, which makes these algorithms unpractical in reality. Keeping this in mind, in this article, we propose an efficient memetic algorithm (MA) for automated convolutional neural network (CNN) architecture search. In contrast to existing EO algorithms for CNN architecture design, a new cell-based architecture search space, and new global and local search operators are proposed for CNN architecture search. To further improve the efficiency of our proposed algorithm, we develop a one-epoch-based performance estimation strategy without any pretrained models to evaluate each found architecture on the training datasets. To investigate the performance of the proposed method, comprehensive empirical studies are conducted against 34 state-of-the-art peer algorithms, including manual algorithms, reinforcement learning (RL) algorithms, gradient-based algorithms, and evolutionary algorithms (EAs), on widely used CIFAR10 and CIFAR100 datasets. The obtained results confirmed the efficacy of the proposed approach for automated CNN architecture design.

Heterogeneous Multi-Party Learning With Data-Driven Network Sampling.

Multi-party learning provides an effective approach for training a machine learning model, e.g., deep neural networks (DNNs), over decentralized data by leveraging multiple decentralized computing devices, subjected to legal and practical constraints. Different parties, so-called local participants, usually provide heterogenous data in a decentralized mode, leading to non-IID data distributions across different local participants which pose a notorious challenge for multi-party learning. To address this challenge, we propose a novel heterogeneous differentiable sampling (HDS) framework. Inspired by the dropout strategy in DNNs, a data-driven network sampling strategy is devised in the HDS framework, with differentiable sampling rates which allow each local participant to extract from a common global model the optimal local model that best adapts to its own data properties so that the size of the local model can be significantly reduced to enable more efficient inference. Meanwhile, co-adaptation of the global model via learning such local models allows for achieving better learning performance under non-IID data distributions and speeds up the convergence of the global model. Experiments have demonstrated the superiority of the proposed method over several popular multi-party learning techniques in the multi-party settings with non-IID data distributions.

Choose Appropriate Subproblems for Collaborative Modeling in Expensive Multiobjective Optimization.

In dealing with the expensive multiobjective optimization problem, some algorithms convert it into a number of single-objective subproblems for optimization. At each iteration, these algorithms conduct surrogate-assisted optimization on one or multiple subproblems. However, these subproblems may be unnecessary or resolved. Operating on such subproblems can cause server inefficiencies, especially in the case of expensive optimization. To overcome this shortcoming, we propose an adaptive subproblem selection (ASS) strategy to identify the most promising subproblems for further modeling. To better leverage the cross information between the subproblems, we use the collaborative multioutput Gaussian process surrogate to model them jointly. Moreover, the commonly used acquisition functions (also known as infill criteria) are investigated in this article. Our analysis reveals that these acquisition functions may cause severe imbalances between exploitation and exploration in multiobjective optimization scenarios. Consequently, we develop a new acquisition function, namely, adaptive lower confidence bound (ALCB), to cope with it. The experimental results on three different sets of benchmark problems indicate that our proposed algorithm is competitive. Beyond that, we also quantitatively validate the effectiveness of the ASS strategy, the CoMOGP model, and the ALCB acquisition function.

Multiparty Dual Learning.

The performance of machine learning algorithms heavily relies on the availability of a large amount of training data. However, in reality, data usually reside in distributed parties such as different institutions and may not be directly gathered and integrated due to various data policy constraints. As a result, some parties may suffer from insufficient data available for training machine learning models. In this article, we propose a multiparty dual learning (MPDL) framework to alleviate the problem of limited data with poor quality in an isolated party. Since the knowledge-sharing processes for multiple parties always emerge in dual forms, we show that dual learning is naturally suitable to handle the challenge of missing data, and explicitly exploits the probabilistic correlation and structural relationship between dual tasks to regularize the training process. We introduce a feature-oriented differential privacy with mathematical proof, in order to avoid possible privacy leakage of raw features in the dual inference process. The approach requires minimal modifications to the existing multiparty learning structure, and each party can build flexible and powerful models separately, whose accuracy is no less than nondistributed self-learning approaches. The MPDL framework achieves significant improvement compared with state-of-the-art multiparty learning methods, as we demonstrated through simulations on real-world datasets.

A Collaborative Multimodal Learning-Based Framework for COVID-19 Diagnosis.

The pandemic of coronavirus disease 2019 (COVID-19) has led to a global public health crisis, which caused millions of deaths and billions of infections, greatly increasing the pressure on medical resources. With the continuous emergence of viral mutations, developing automated tools for COVID-19 diagnosis is highly desired to assist the clinical diagnosis and reduce the tedious workload of image interpretation. However, medical images in a single site are usually of a limited amount or weakly labeled, while integrating data scattered around different institutions to build effective models is not allowed due to data policy restrictions. In this article, we propose a novel privacy-preserving cross-site framework for COVID-19 diagnosis with multimodal data, seeking to effectively leverage heterogeneous data from multiple parties while preserving patients' privacy. Specifically, a Siamese branched network is introduced as the backbone to capture inherent relationships across heterogeneous samples. The redesigned network is capable of handling semisupervised inputs in multimodalities and conducting task-specific training, in order to improve the model performance of various scenarios. The framework achieves significant improvement compared with state-of-the-art methods, as we demonstrate through extensive simulations on real-world datasets.

Pareto optimization with small data by learning across common objective spaces.

In multi-objective optimization, it becomes prohibitively difficult to cover the Pareto front (PF) as the number of points scales exponentially with the dimensionality of the objective space. The challenge is exacerbated in expensive optimization domains where evaluation data is at a premium. To overcome insufficient representations of PFs, Pareto estimation (PE) invokes inverse machine learning to map preferred but unexplored regions along the front to the Pareto set in decision space. However, the accuracy of the inverse model depends on the training data, which is inherently scarce/small given high-dimensional/expensive objectives. To alleviate this small data challenge, this paper marks a first study on multi-source inverse transfer learning for PE. A method to maximally utilize experiential source tasks to augment PE in the target optimization task is proposed. Information transfers between heterogeneous source-target pairs is uniquely enabled in the inverse setting through the unification provided by common objective spaces. Our approach is tested experimentally on benchmark functions as well as on high-fidelity, multidisciplinary simulation data of composite materials manufacturing processes, revealing significant gains to the predictive accuracy and PF approximation capacity of Pareto set learning. With such accurate inverse models made feasible, a future of on-demand human-machine interaction facilitating multi-objective decisions is envisioned.

DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation.

Recently, one critical issue looms large in the field of recommender systems - there are no effective benchmarks for rigorous evaluation - which consequently leads to unreproducible evaluation and unfair comparison. We, therefore, conduct studies from the perspectives of practical theory and experiments, aiming at benchmarking recommendation for rigorous evaluation. Regarding the theoretical study, a series of hyper-factors affecting recommendation performance throughout the whole evaluation chain are systematically summarized and analyzed via an exhaustive review on 141 papers published at eight top-tier conferences within 2017-2020. We then classify them into model-independent and model-dependent hyper-factors, and different modes of rigorous evaluation are defined and discussed in-depth accordingly. For the experimental study, we release DaisyRec 2.0 library by integrating these hyper-factors to perform rigorous evaluation, whereby a holistic empirical study is conducted to unveil the impacts of different hyper-factors on recommendation performance. Supported by the theoretical and experimental studies, we finally create benchmarks for rigorous evaluation by proposing standardized procedures and providing performance of ten state-of-the-arts across six evaluation metrics on six datasets as a reference for later study. Overall, our work sheds light on the issues in recommendation evaluation, provides potential solutions for rigorous evaluation, and lays foundation for further investigation.

Scaling Multiobjective Evolution to Large Data With Minions: A Bayes-Informed Multitask Approach.

In an era of pervasive digitalization, the growing volume and variety of data streams poses a new challenge to the efficient running of data-driven optimization algorithms. Targeting scalable multiobjective evolution under large-instance data, this article proposes the general idea of using subsampled small-data tasks as helpful minions (i.e., auxiliary source tasks) to quickly optimize for large datasets-via an evolutionary multitasking framework. Within this framework, a novel computational resource allocation strategy is designed to enable the effective utilization of the minions while guarding against harmful negative transfers. To this end, an intertask empirical correlation measure is defined and approximated via Bayes' rule, which is then used to allocate resources online in proportion to the inferred degree of source-target correlation. In the experiments, the performance of the proposed algorithm is verified on: 1) sample average approximations of benchmark multiobjective optimization problems under uncertainty and 2) practical multiobjective hyperparameter tuning of deep neural network models. The results show that the proposed algorithm can obtain up to about 73% speedup relative to existing approaches, demonstrating its ability to efficiently tackle real-world multiobjective optimization involving evaluations on large datasets.