Toward Explainable Multiparty Learning: A Contrastive Knowledge Sharing Framework.

Multiparty learning provides solutions for training joint models with decentralized data under legal and practical constraints. However, traditional multiparty learning approaches are confronted with obstacles, such as system heterogeneity, statistical heterogeneity, and incentive design. Determining how to deal with these challenges and further improve the efficiency and performance of multiparty learning has become an urgent problem to be solved. In this article, we propose a novel contrastive multiparty learning framework for knowledge refinement and sharing with an accountable incentive mechanism. Since the existing parameter averaging method is contradictory to the learning paradigm of neural networks, we simulate the process of human cognition and communication and analogize multiparty learning as a many-to-one knowledge-sharing problem. The approach is capable of integrating the acquired explicit knowledge of each client in a transparent manner without privacy disclosure, and it reduces the dependence on data distribution and communication environments. The proposed scheme achieves significant improvement in model performance in a variety of scenarios, as we demonstrated through experiments on several real-world datasets.

Deep-Growing Neural Network With Manifold Constraints for Hyperspectral Image Classification.

In the absence of sufficient labels, deep neural networks (DNNs) are prone to overfitting, resulting in poor performance and difficulty in training. Thus, many semisupervised methods aim to use unlabeled sample information to compensate for the lack of label quantity. However, as the available pseudolabels increase, the fixed structure of traditional models has difficulty in matching them, limiting their effectiveness. Therefore, a deep-growing neural network with manifold constraints (DGNN-MC) is proposed. It can deepen the corresponding network structure with the expansion of a high-quality pseudolabel pool and preserve the local structure between the original and high-dimensional data in semisupervised learning. First, the framework filters the output of the shallow network to obtain pseudolabeled samples with high confidence and adds them to the original training set to form a new pseudolabeled training set. Second, according to the size of the new training set, it increases the depth of the layers to obtain a deeper network and conducts the training. Finally, it obtains new pseudolabeled samples and deepens the layers again until the network growth is completed. The growing model proposed in this article can be applied to other multilayer networks, as their depth can be transformed. Taking HSI classification as an example, a natural semisupervised problem, the experimental results demonstrate the superiority and effectiveness of our method, which can mine more reliable information for better utilization and fully balance the growing amount of labeled data and network learning ability.

A Collaborative Multimodal Learning-Based Framework for COVID-19 Diagnosis.

The pandemic of coronavirus disease 2019 (COVID-19) has led to a global public health crisis, which caused millions of deaths and billions of infections, greatly increasing the pressure on medical resources. With the continuous emergence of viral mutations, developing automated tools for COVID-19 diagnosis is highly desired to assist the clinical diagnosis and reduce the tedious workload of image interpretation. However, medical images in a single site are usually of a limited amount or weakly labeled, while integrating data scattered around different institutions to build effective models is not allowed due to data policy restrictions. In this article, we propose a novel privacy-preserving cross-site framework for COVID-19 diagnosis with multimodal data, seeking to effectively leverage heterogeneous data from multiple parties while preserving patients' privacy. Specifically, a Siamese branched network is introduced as the backbone to capture inherent relationships across heterogeneous samples. The redesigned network is capable of handling semisupervised inputs in multimodalities and conducting task-specific training, in order to improve the model performance of various scenarios. The framework achieves significant improvement compared with state-of-the-art methods, as we demonstrate through extensive simulations on real-world datasets.

RORNet: Partial-to-Partial Registration Network With Reliable Overlapping Representations.

Three-dimensional point cloud registration is an important field in computer vision. Recently, due to the increasingly complex scenes and incomplete observations, many partial-overlap registration methods based on overlap estimation have been proposed. These methods heavily rely on the extracted overlapping regions with their performances greatly degraded when the overlapping region extraction underperforms. To solve this problem, we propose a partial-to-partial registration network (RORNet) to find reliable overlapping representations from the partially overlapping point clouds and use these representations for registration. The idea is to select a small number of key points called reliable overlapping representations from the estimated overlapping points, reducing the side effect of overlap estimation errors on registration. Although it may filter out some inliers, the inclusion of outliers has a much bigger influence than the omission of inliers on the registration task. The RORNet is composed of overlapping points' estimation module and representations' generation module. Different from the previous methods of direct registration after extraction of overlapping areas, RORNet adds the step of extracting reliable representations before registration, where the proposed similarity matrix downsampling method is used to filter out the points with low similarity and retain reliable representations, and thus reduce the side effects of overlap estimation errors on the registration. Besides, compared with previous similarity-based and score-based overlap estimation methods, we use the dual-branch structure to combine the benefits of both, which is less sensitive to noise. We perform overlap estimation experiments and registration experiments on the ModelNet40 dataset, outdoor large scene dataset KITTI, and natural data Stanford Bunny dataset. The experimental results demonstrate that our method is superior to other partial registration methods. Our code is available at https://github.com/superYuezhang/RORNet.

Device-Performance-Driven Heterogeneous Multiparty Learning for Arbitrary Images.

Multiparty learning (MPL) is an emerging framework for privacy-preserving collaborative learning. It enables individual devices to build a knowledge-shared model and remaining sensitive data locally. However, with the continuous increase of users, the heterogeneity gap between data and equipment becomes wider, which leads to the problem of model heterogeneous. In this article, we concentrate on two practical issues: data heterogeneous problem and model heterogeneous problem, and propose a novel personal MPL method named device-performance-driven heterogeneous MPL (HMPL). First, facing the data heterogeneous problem, we focus on the problem of various devices holding arbitrary data sizes. We introduce a heterogeneous feature-map integration method to adaptively unify the various feature maps. Meanwhile, to handle the model heterogeneous problem, as it is essential to customize models for adapting to the various computing performances, we propose a layer-wise model generation and aggregation strategy. The method can generate customized models based on the device's performance. In the aggregation process, the shared model parameters are updated through the rules that the network layers with the same semantics are aggregated with each other. Extensive experiments are conducted on four popular datasets, and the result demonstrates that our proposed framework outperforms the state of the art (SOTA).

Heterogeneous Multi-Party Learning With Data-Driven Network Sampling.

Multi-party learning provides an effective approach for training a machine learning model, e.g., deep neural networks (DNNs), over decentralized data by leveraging multiple decentralized computing devices, subjected to legal and practical constraints. Different parties, so-called local participants, usually provide heterogenous data in a decentralized mode, leading to non-IID data distributions across different local participants which pose a notorious challenge for multi-party learning. To address this challenge, we propose a novel heterogeneous differentiable sampling (HDS) framework. Inspired by the dropout strategy in DNNs, a data-driven network sampling strategy is devised in the HDS framework, with differentiable sampling rates which allow each local participant to extract from a common global model the optimal local model that best adapts to its own data properties so that the size of the local model can be significantly reduced to enable more efficient inference. Meanwhile, co-adaptation of the global model via learning such local models allows for achieving better learning performance under non-IID data distributions and speeds up the convergence of the global model. Experiments have demonstrated the superiority of the proposed method over several popular multi-party learning techniques in the multi-party settings with non-IID data distributions.

Multiparty Dual Learning.

The performance of machine learning algorithms heavily relies on the availability of a large amount of training data. However, in reality, data usually reside in distributed parties such as different institutions and may not be directly gathered and integrated due to various data policy constraints. As a result, some parties may suffer from insufficient data available for training machine learning models. In this article, we propose a multiparty dual learning (MPDL) framework to alleviate the problem of limited data with poor quality in an isolated party. Since the knowledge-sharing processes for multiple parties always emerge in dual forms, we show that dual learning is naturally suitable to handle the challenge of missing data, and explicitly exploits the probabilistic correlation and structural relationship between dual tasks to regularize the training process. We introduce a feature-oriented differential privacy with mathematical proof, in order to avoid possible privacy leakage of raw features in the dual inference process. The approach requires minimal modifications to the existing multiparty learning structure, and each party can build flexible and powerful models separately, whose accuracy is no less than nondistributed self-learning approaches. The MPDL framework achieves significant improvement compared with state-of-the-art multiparty learning methods, as we demonstrated through simulations on real-world datasets.

Self-Paced Co-Training of Graph Neural Networks for Semi-Supervised Node Classification.

Graph neural networks (GNNs) have demonstrated great success in many graph data-based applications. The impressive behavior of GNNs typically relies on the availability of a sufficient amount of labeled data for model training. However, in practice, obtaining a large number of annotations is prohibitively labor-intensive and even impossible. Co-training is a popular semi-supervised learning (SSL) paradigm, which trains multiple models based on a common training set while augmenting the limited amount of labeled data used for training each model via the pseudolabeled data generated from the prediction results of other models. Most of the existing co-training works do not control the quality of pseudolabeled data when using them. Therefore, the inaccurate pseudolabels generated by immature models in the early stage of the training process are likely to cause noticeable errors when they are used for augmenting the training data for other models. To address this issue, we propose a self-paced co-training for the GNN (SPC-GNN) framework for semi-supervised node classification. This framework trains multiple GNNs with the same or different structures on different representations of the same training data. Each GNN carries out SSL by using both the originally available labeled data and the augmented pseudolabeled data generated from other GNNs. To control the quality of pseudolabels, a self-paced label augmentation strategy is designed to make the pseudolabels generated at a higher confidence level to be utilized earlier during training such that the negative impact of inaccurate pseudolabels on training data augmentation, and accordingly, the subsequent training process can be mitigated. Finally, each of the trained GNN is evaluated on a validation set, and the best-performing one is chosen as the output. To improve the training effectiveness of the framework, we devise a pretraining followed by a two-step optimization scheme to train GNNs. Experimental results on the node classification task demonstrate that the proposed framework achieves significant improvement over the state-of-the-art SSL methods.

Semisupervised Graph Neural Networks for Graph Classification.

Graph classification aims to predict the label associated with a graph and is an important graph analytic task with widespread applications. Recently, graph neural networks (GNNs) have achieved state-of-the-art results on purely supervised graph classification by virtue of the powerful representation ability of neural networks. However, almost all of them ignore the fact that graph classification usually lacks reasonably sufficient labeled data in practical scenarios due to the inherent labeling difficulty caused by the high complexity of graph data. The existing semisupervised GNNs typically focus on the task of node classification and are incapable to deal with graph classification. To tackle the challenging but practically useful scenario, we propose a novel and general semisupervised GNN framework for graph classification, which takes full advantage of a slight amount of labeled graphs and abundant unlabeled graph data. In our framework, we train two GNNs as complementary views for collaboratively learning high-quality classifiers using both labeled and unlabeled graphs. To further exploit the view itself, we constantly select pseudo-labeled graph examples with high confidence from its own view for enlarging the labeled graph dataset and enhancing predictions on graphs. Furthermore, the proposed framework is investigated on two specific implementation regimes with a few labeled graphs and the extremely few labeled graphs, respectively. Extensive experimental results demonstrate the effectiveness of our proposed semisupervised GNN framework for graph classification on several benchmark datasets.

DANet: Semi-supervised differentiated auxiliaries guided network for video action recognition.

Video Action Recognition (ViAR) aims to identify the category of the human action observed in a given video. With the advent of Deep Learning (DL) techniques, noticeable performance breakthroughs have been achieved in this study. However, the success of most existing DL-based ViAR methods heavily relies on the existence of a large amount of annotated data, i.e., videos with corresponding action categories. In practice, obtaining such a desired number of annotations is often difficult due to expensive labeling costs, which may lead to significant performance degradation for these methods. To address this issue, we propose an end-to-end semi-supervised Differentiated Auxiliary guided Network (DANet) to best use a few annotated videos. Except for the common supervised learning on a few annotated videos, the DANet also involves the knowledge of multiple pre-trained auxiliary networks to optimize the ViAR network in a self-supervised way on the unannotated data by removing the annotations. Considering the tight connection between video action recognition and classical static image-based visual tasks, the abundant knowledge from the pre-trained static image-based models can be used for training the ViAR model. Specifically, the DANet is a two-branch architecture, which includes a target branch of the ViAR network, and an auxiliary branch of multiple auxiliary networks (i.e., referring to diverse off-the-shelf models of relevant image tasks). Given a limited number of annotated videos, we train the target ViAR network end-to-end in a semi-supervised way, namely, with both the supervised cross-entropy loss on annotated videos, and the per-auxiliary weighted self-supervised contrastive losses on the same videos but without using annotations. Besides, we further explore different weighted guidance of the auxiliary networks to the ViAR network to better reflect different relationships between the image-based models and the ViAR model. Finally, we conduct extensive experiments on several popular action recognition benchmarks in comparison with existing state-of-the-art methods, and the experimental results demonstrate the superiority of DANet over most of the compared methods. In particular, the DANet obviously suppresses state-of-the-art ViAR methods even with very fewer annotated videos.

A Multifactorial Optimization Framework Based on Adaptive Intertask Coordinate System.

The searching ability of the population-based search algorithms strongly relies on the coordinate system on which they are implemented. However, the widely used coordinate systems in the existing multifactorial optimization (MFO) algorithms are still fixed and might not be suitable for various function landscapes with differential modalities, rotations, and dimensions; thus, the intertask knowledge transfer might not be efficient. Therefore, this article proposes a novel intertask knowledge transfer strategy for MFOs implemented upon an active coordinate system that is established on a common subspace of two search spaces. The proper coordinate system might identify some common modality in a proper subspace to some extent. In this article, to seek the intermediate subspace, we innovatively introduce the geodesic flow that starts from a subspace, reaching another subspace in unit time. A low-dimension intermediate subspace is drawn from a uniform distribution defined on the geodesic flow, and the corresponding coordinate system is given. The intertask trial generation method is applied to the individuals by first projecting them on the low-dimension subspace, which reveals the important invariant features of the multiple function landscapes. Since intermediate subspace is generated from the major eigenvectors of tasks' spaces, this model turns out to be intrinsically regularized by neglecting the minor and small eigenvalues. Therefore, the transfer strategy can alleviate the influence of noise led by redundant dimensions. The proposed method exhibits promising performance in the experiments.

Commonality Autoencoder: Learning Common Features for Change Detection From Heterogeneous Images.

Change detection based on heterogeneous images, such as optical images and synthetic aperture radar images, is a challenging problem because of their huge appearance differences. To combat this problem, we propose an unsupervised change detection method that contains only a convolutional autoencoder (CAE) for feature extraction and the commonality autoencoder for commonalities exploration. The CAE can eliminate a large part of redundancies in two heterogeneous images and obtain more consistent feature representations. The proposed commonality autoencoder has the ability to discover common features of ground objects between two heterogeneous images by transforming one heterogeneous image representation into another. The unchanged regions with the same ground objects share much more common features than the changed regions. Therefore, the number of common features can indicate changed regions and unchanged regions, and then a difference map can be calculated. At last, the change detection result is generated by applying a segmentation algorithm to the difference map. In our method, the network parameters of the commonality autoencoder are learned by the relevance of unchanged regions instead of the labels. Our experimental results on five real data sets demonstrate the promising performance of the proposed framework compared with several existing approaches.

Locality preserving dense graph convolutional networks with graph context-aware node representations.

Graph convolutional networks (GCNs) have been widely used for representation learning on graph data, which can capture structural patterns on a graph via specifically designed convolution and readout operations. In many graph classification applications, GCN-based approaches have outperformed traditional methods. However, most of the existing GCNs are inefficient to preserve local information of graphs - a limitation that is especially problematic for graph classification. In this work, we propose a locality-preserving dense GCN with graph context-aware node representations. Specifically, our proposed model incorporates a local node feature reconstruction module to preserve initial node features into node representations, which is realized via a simple but effective encoder-decoder mechanism. To capture local structural patterns in neighborhoods representing different ranges of locality, dense connectivity is introduced to connect each convolutional layer and its corresponding readout with all previous convolutional layers. To enhance node representativeness, the output of each convolutional layer is concatenated with the output of the previous layer's readout to form a global context-aware node representation. In addition, a self-attention module is introduced to aggregate layer-wise representations to form the final graph-level representation. Experiments on benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art methods in terms of classification accuracy.

Evolving Deep Neural Networks via Cooperative Coevolution With Backpropagation.

Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.

Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition.

In recent years, deep learning achieves remarkable results in the field of artificial intelligence. However, the training process of deep neural networks may cause the leakage of individual privacy. Given the model and some background information of the target individual, the adversary can maliciously infer the sensitive feature of the target individual. Therefore, it is imperative to preserve the sensitive information in the training data. Differential privacy is a state-of-the-art paradigm for providing the privacy guarantee of datasets, which protects the private and sensitive information from the attack of adversaries significantly. However, the existing privacy-preserving models based on differential privacy are less than satisfactory since traditional approaches always inject the same amount of noise into parameters to preserve the sensitive information, which may impact the trade-off between the model utility and the privacy guarantee of training data. In this paper, we present a general differentially private deep neural networks learning framework based on relevance analysis, which aims to bridge the gap between private and non-private models while providing an effective privacy guarantee of sensitive information. The proposed model perturbs gradients according to the relevance between neurons in different layers and the model output. Specifically, during the process of backward propagation, more noise is added to gradients of neurons that have less relevance to the model output, and vice-versa. Experiments on five real datasets demonstrate that our mechanism not only bridges the gap between private and non-private models, but also prevents the disclosure of sensitive information effectively.

Bipartite Differential Neural Network for Unsupervised Image Change Detection.

Image change detection detects the regions of change in multiple images of the same scene taken at different times, which plays a crucial role in many applications. The two most popular image change detection techniques are as follows: pixel-based methods heavily rely on accurate image coregistration while object-based approaches can tolerate coregistration errors to some extent but are sensitive to image segmentation or classification errors. To address these issues, we propose an unsupervised image change detection approach based on a novel bipartite differential neural network (BDNN). The BDNN is a deep neural network with two input ends, which can extract the holistic features from the unchanged regions in the two input images, where two learnable change disguise maps (CDMs) are used to disguise the changed regions in the two input images, respectively, and thus demarcate the unchanged regions therein. The network parameters and CDMs will be learned by optimizing an objective function, which combines a loss function defined as the likelihood of the given input image pair over all possible input image pairs and two constraints imposed on CDMs. Compared with the pixel-based and object-based techniques, the BDNN is less sensitive to inaccurate image coregistration and does not involve image segmentation or classification. In fact, it can even skip over coregistration if the degree of transformation (due to the different view angles and/or positions of the camera) between the two input images is not that large. We compare the proposed approach with several state-of-the-art image change detection methods on various homogeneous and heterogeneous image pairs with and without coregistration. The results demonstrate the superiority of the proposed approach.

A Multi-Task Representation Learning Architecture for Enhanced Graph Classification.

Composed of nodes and edges, graph structured data are organized in the non-Euclidean geometric space and ubiquitous especially in chemical compounds, proteins, etc. They usually contain rich structure information, and how to effectively extract inherent features of them is of great significance on the determination of function or traits in medicine and biology. Recently, there is a growing interest in learning graph-level representations for graph classification. Existing graph classification strategies based on graph neural networks broadly follow a single-task learning framework and manage to learn graph-level representations through aggregating node-level representations. However, they lack the efficient utilization of labels of nodes in a graph. In this paper, we propose a novel multi-task representation learning architecture coupled with the task of supervised node classification for enhanced graph classification. Specifically, the node classification task enforces node-level representations to take full advantage of node labels available in the graph and the graph classification task allows for learning graph-level representations in an end-to-end manner. Experimental results on multiple benchmark datasets demonstrate that the proposed architecture performs significantly better than various single-task graph neural network methods for graph classification.

Evolutionary Multitasking via Explicit Autoencoding.

Evolutionary multitasking (EMT) is an emerging research topic in the field of evolutionary computation. In contrast to the traditional single-task evolutionary search, EMT conducts evolutionary search on multiple tasks simultaneously. It aims to improve convergence characteristics across multiple optimization problems at once by seamlessly transferring knowledge among them. Due to the efficacy of EMT, it has attracted lots of research attentions and several EMT algorithms have been proposed in the literature. However, existing EMT algorithms are usually based on a common mode of knowledge transfer in the form of implicit genetic transfer through chromosomal crossover. This mode cannot make use of multiple biases embedded in different evolutionary search operators, which could give better search performance when properly harnessed. Keeping this in mind, this paper proposes an EMT algorithm with explicit genetic transfer across tasks, namely EMT via autoencoding, which allows the incorporation of multiple search mechanisms with different biases in the EMT paradigm. To confirm the efficacy of the proposed EMT algorithm with explicit autoencoding, comprehensive empirical studies have been conducted on both the single- and multi-objective multitask optimization problems.

Multiobjective Deep Belief Networks Ensemble for Remaining Useful Life Estimation in Prognostics.

In numerous industrial applications where safety, efficiency, and reliability are among primary concerns, condition-based maintenance (CBM) is often the most effective and reliable maintenance policy. Prognostics, as one of the key enablers of CBM, involves the core task of estimating the remaining useful life (RUL) of the system. Neural networks-based approaches have produced promising results on RUL estimation, although their performances are influenced by handcrafted features and manually specified parameters. In this paper, we propose a multiobjective deep belief networks ensemble (MODBNE) method. MODBNE employs a multiobjective evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives. The eventually evolved DBNs are combined to establish an ensemble model used for RUL estimation, where combination weights are optimized via a single-objective differential evolution algorithm using a task-oriented objective function. We evaluate the proposed method on several prognostic benchmarking data sets and also compare it with some existing approaches. Experimental results demonstrate the superiority of our proposed method.

Multivariate image segmentation using semantic region growing with adaptive edge penalty.

Multivariate image segmentation is a challenging task, influenced by large intraclass variation that reduces class distinguishability as well as increased feature space sparseness and solution space complexity that impose computational cost and degrade algorithmic robustness. To deal with these problems, a Markov random field (MRF) based multivariate segmentation algorithm called "multivariate iterative region growing using semantics" (MIRGS) is presented. In MIRGS, the impact of intraclass variation and computational cost are reduced using the MRF spatial context model incorporated with adaptive edge penalty and applied to regions. Semantic region growing starting from watershed over-segmentation and performed alternatively with segmentation gradually reduces the solution space size, which improves segmentation effectiveness. As a multivariate iterative algorithm, MIRGS is highly sensitive to initial conditions. To suppress initialization sensitivity, it employs a region-level k -means (RKM) based initialization method, which consistently provides accurate initial conditions at low computational cost. Experiments show the superiority of RKM relative to two commonly used initialization methods. Segmentation tests on a variety of synthetic and natural multivariate images demonstrate that MIRGS consistently outperforms three other published algorithms.