Episodic task agnostic contrastive training for multi-task learning.

Learning knowledge from different tasks to improve the general learning performance is crucial for designing an efficient algorithm. In this work, we tackle the Multi-task Learning (MTL) problem, where the learner extracts the knowledge from different tasks simultaneously with limited data. Previous works have been designing the MTL models by taking advantage of the transfer learning techniques, requiring the knowledge of the task index, which is not realistic in many practical scenarios. In contrast, we consider the scenario that the task index is not explicitly known, under which the features extracted by the neural networks are task agnostic. To learn the task agnostic invariant features, we implement model agnostic meta-learning by leveraging the episodic training scheme to capture the common features across tasks. Apart from the episodic training scheme, we further implemented a contrastive learning objective to improve the feature compactness for a better prediction boundary in the embedding space. We conduct extensive experiments on several benchmarks compared with several recent strong baselines to demonstrate the effectiveness of the proposed method. The results showed that our method provides a practical solution for real-world scenarios, where the task index is agnostic to the learner and can outperform several strong baselines, achieving state-of-the-art performances.

On the benefits of representation regularization in invariance based domain generalization.

A crucial aspect of reliable machine learning is to design a deployable system for generalizing new related but unobserved environments. Domain generalization aims to alleviate such a prediction gap between the observed and unseen environments. Previous approaches commonly incorporated learning the invariant representation for achieving good empirical performance. In this paper, we reveal that merely learning the invariant representation is vulnerable to the related unseen environment. To this end, we derive a novel theoretical analysis to control the unseen test environment error in the representation learning, which highlights the importance of controlling the smoothness of representation. In practice, our analysis further inspires an efficient regularization method to improve the robustness in domain generalization. The proposed regularization is orthogonal to and can be straightforwardly adopted in existing domain generalization algorithms that ensure invariant representation learning. Empirical results show that our algorithm outperforms the base versions in various datasets and invariance criteria.

Task Similarity Estimation Through Adversarial Multitask Neural Network.

Multitask learning (MTL) aims at solving the related tasks simultaneously by exploiting shared knowledge to improve performance on individual tasks. Though numerous empirical results supported the notion that such shared knowledge among tasks plays an essential role in MTL, the theoretical understanding of the relationships between tasks and their impact on learning shared knowledge is still an open problem. In this work, we are developing a theoretical perspective of the benefits involved in using information similarity for MTL. To this end, we first propose an upper bound on the generalization error by implementing the Wasserstein distance as the similarity metric. This indicates the practical principles of applying the similarity information to control the generalization errors. Based on those theoretical results, we revisited the adversarial multitask neural network and proposed a new training algorithm to learn the task relation coefficients and neural network parameters automatically. The computer vision benchmarks reveal the abilities of the proposed algorithms to improve the empirical performance. Finally, we test the proposed approach on real medical data sets, showing its advantage for extracting task relations.

Common Spatial Pattern Reformulated for Regularizations in Brain-Computer Interfaces.

Common spatial pattern (CSP) is one of the most successful feature extraction algorithms for brain-computer interfaces (BCIs). It aims to find spatial filters that maximize the projected variance ratio between the covariance matrices of the multichannel electroencephalography (EEG) signals corresponding to two mental tasks, which can be formulated as a generalized eigenvalue problem (GEP). However, it is challenging in principle to impose additional regularization onto the CSP to obtain structural solutions (e.g., sparse CSP) due to the intrinsic nonconvexity and invariance property of GEPs. This article reformulates the CSP as a constrained minimization problem and establishes the equivalence of the reformulated and the original CSPs. An efficient algorithm is proposed to solve this optimization problem by alternately performing singular value decomposition (SVD) and least squares. Under this new formulation, various regularization techniques for linear regression can then be easily implemented to regularize the CSPs for different learning paradigms, such as the sparse CSP, the transfer CSP, and the multisubject CSP. Evaluations on three BCI competition datasets show that the regularized CSP algorithms outperform other baselines, especially for the high-dimensional small training set. The extensive results validate the efficiency and effectiveness of the proposed CSP formulation in different learning contexts.