Few-Shot Continual Learning via Flat-to-Wide Approaches.

The existing approaches on continual learning (CL) call for a lot of samples in their training processes. Such approaches are impractical for many real-world problems having limited samples because of the overfitting problem. This article proposes a few-shot CL approach, termed flat-to-wide approach (FLOWER), where a flat-to-wide learning process finding the flat-wide minima is proposed to address the catastrophic forgetting (CF) problem. The issue of data scarcity is overcome with a data augmentation approach making use of a ball-generator concept to restrict the sampling space into the smallest enclosing ball. Our numerical studies demonstrate the advantage of FLOWER achieving significantly improved performances over prior arts notably in the small base tasks. For further study, source codes of FLOWER, competitor algorithms, and experimental logs are shared publicly in https://github.com/anwarmaxsum/FLOWER.

Pareto optimization with small data by learning across common objective spaces.

In multi-objective optimization, it becomes prohibitively difficult to cover the Pareto front (PF) as the number of points scales exponentially with the dimensionality of the objective space. The challenge is exacerbated in expensive optimization domains where evaluation data is at a premium. To overcome insufficient representations of PFs, Pareto estimation (PE) invokes inverse machine learning to map preferred but unexplored regions along the front to the Pareto set in decision space. However, the accuracy of the inverse model depends on the training data, which is inherently scarce/small given high-dimensional/expensive objectives. To alleviate this small data challenge, this paper marks a first study on multi-source inverse transfer learning for PE. A method to maximally utilize experiential source tasks to augment PE in the target optimization task is proposed. Information transfers between heterogeneous source-target pairs is uniquely enabled in the inverse setting through the unification provided by common objective spaces. Our approach is tested experimentally on benchmark functions as well as on high-fidelity, multidisciplinary simulation data of composite materials manufacturing processes, revealing significant gains to the predictive accuracy and PF approximation capacity of Pareto set learning. With such accurate inverse models made feasible, a future of on-demand human-machine interaction facilitating multi-objective decisions is envisioned.

Unsupervised Continual Learning in Streaming Environments.

A deep clustering network (DCN) is desired for data streams because of its aptitude in extracting natural features thus bypassing the laborious feature engineering step. While automatic construction of deep networks in streaming environments remains an open issue, it is also hindered by the expensive labeling cost of data streams rendering the increasing demand for unsupervised approaches. This article presents an unsupervised approach of DCN construction on the fly via simultaneous deep learning and clustering termed autonomous DCN (ADCN). It combines the feature extraction layer and autonomous fully connected layer in which both network width and depth are self-evolved from data streams based on the bias-variance decomposition of reconstruction loss. The self-clustering mechanism is performed in the deep embedding space of every fully connected layer, while the final output is inferred via the summation of cluster prediction score. Furthermore, a latent-based regularization is incorporated to resolve the catastrophic forgetting issue. A rigorous numerical study has shown that ADCN produces better performance compared with its counterparts while offering fully autonomous construction of ADCN structure in streaming environments in the absence of any labeled samples for model updates. To support the reproducible research initiative, codes, supplementary material, and raw results of ADCN are made available in https://github.com/andriash001/AutonomousDCN.git.

Autonomous Cross Domain Adaptation Under Extreme Label Scarcity.

A cross domain multistream classification is a challenging problem calling for fast domain adaptations to handle different but related streams in never-ending and rapidly changing environments. Notwithstanding that existing multistream classifiers assume no labeled samples in the target stream, they still incur expensive labeling costs since they require fully labeled samples of the source stream. This article aims to attack the problem of extreme label shortage in the cross domain multistream classification problems where only very few labeled samples of the source stream are provided before process runs. Our solution, namely, Learning Streaming Process from Partial Ground Truth (LEOPARD), is built upon a flexible deep clustering network where its hidden nodes, layers, and clusters are added and removed dynamically with respect to varying data distributions. A deep clustering strategy is underpinned by a simultaneous feature learning and clustering technique leading to clustering-friendly latent spaces. A domain adaptation strategy relies on the adversarial domain adaptation technique where a feature extractor is trained to fool a domain classifier by classifying source and target streams. Our numerical study demonstrates the efficacy of LEOPARD where it delivers improved performances compared to prominent algorithms in 15 of 24 cases. Source codes of LEOPARD are shared in https://github.com/wengweng001/LEOPARD.git to enable further study.

Subject-independent Classification on Brain-Computer Interface using Autonomous Deep Learning for finger movement recognition.

The degradation of the subject-independent classification on a brain-computer interface is a challenging issue. One method mostly taken to overcome this problem is by collecting as many subjects as possible and then training the system across all subjects. This article introduces streaming online learning called autonomous deep learning (ADL) to classify five individual fingers based on electroencephalography (EEG) signals to overcome the issue above. ADL is a deep learning architecture that can construct its structure by itself through streaming learning and adapt its structure to the changes occurring in the input. In this article, the input of ADL is a common spatial pattern (CSP) extracted from the EEG signal of healthy subjects. The experimental results on the subject-dependence classification across four subjects using 5fold cross-validation show that that ADL achieved the classification accuracy of around 77%. This performance was excellent compared to a random forest (RF) and a convolutional neural network (CNN). They achieved accuracies of about 53% and 72%, respectively. On the subject-independent classification, ADL outperforms CNN by resulting stable accuracies for both training and testing, different from CNN that experience accuracy degradation to approximately 50%. These results imply that ADL is a promising machine learning in dealing with the issue in the subject-independent classification.

Metaheuristic Spatial Transformation (MST) for accurate detection of Attention Deficit Hyperactivity Disorder (ADHD) using rs-fMRI.

Accurate detection of neuro-psychological disorders such as Attention Deficit Hyperactivity Disorder (ADHD) using resting state functional Magnetic Resonance Imaging (rs-fMRI) is challenging due to high dimensionality of input features, low inter-class separability, small sample size and high intra-class variability. For automatic diagnosis of ADHD and autism, spatial transformation methods have gained significance and have achieved improved classification performance. However, they are not reliable due to lack of generalization in dataset like ADHD with high variance and small sample size. Therefore, in this paper, we present a Metaheuristic Spatial Transformation (MST) approach to convert the spatial filter design problem into a constraint optimization problem, and obtain the solution using a hybrid genetic algorithm. Highly separable features obtained from the MST along with meta-cognitive radial basis function based classifier are utilized to accurately classify ADHD. The performance was evaluated using the ADHD200 consortium dataset using a ten fold cross validation. The results indicate that the MST based classifier produces state of the art classification accuracy of 72.10% (1.71% improvement over previous transformation based methods). Moreover, using MST based classifier the training and testing specificity increased significantly over previous methods in literature. These results clearly indicate that MST enables the determination of the highly discriminant transformation in dataset with high variability, small sample size and large number of features. Further, the performance on the ADHD200 dataset shows that MST based classifier can be reliably used for the accurate diagnosis of ADHD using rs-fMRI.Clinical relevance- Metaheuristic Spatial Transformation (MST) enables reliable and accurate detection of neuropsychological disorders like ADHD from rs-fMRI data characterized by high variability, small sample size and large number of features.

Online Tool Condition Monitoring Based on Parsimonious Ensemble.

Existing methodologies for tool condition monitoring (TCM) still rely on batch approaches which cannot cope with a fast sampling rate of a metal cutting process. Furthermore, they require a retraining process to be completed from scratch when dealing with a new set of machining parameters. This paper presents an online TCM approach based on Parsimonious Ensemble+ (pENsemble+). The unique feature of pENsemble+ lies in its highly flexible principle where both the ensemble structure and base-classifier structure can automatically grow and shrink on the fly based on the characteristics of data streams. Moreover, the online feature selection scenario is integrated to actively sample relevant input attributes. This paper presents advancement of a newly developed ensemble learning algorithm, pENsemble, where the online active learning scenario is incorporated to reduce the operator's labeling effort. The ensemble merging scenario is proposed which allows reduction of ensemble complexity while retaining its diversity. Experimental studies utilizing two real-world manufacturing data streams: 1) metal turning and 2) 3-D-printing processes and comparisons with well-known algorithms were carried out. Furthermore, the efficacy of pENsemble+ was examined using benchmark concept drift data streams. It has been found that pENsemble+ incurs low structural complexity and results in a significant reduction of the operator's labeling effort.

Financial time series forecasting using twin support vector regression.

Financial time series forecasting is a crucial measure for improving and making more robust financial decisions throughout the world. Noisy data and non-stationarity information are the two key factors in financial time series prediction. This paper proposes twin support vector regression for financial time series prediction to deal with noisy data and nonstationary information. Various interesting financial time series datasets across a wide range of industries, such as information technology, the stock market, the banking sector, and the oil and petroleum sector, are used for numerical experiments. Further, to test the accuracy of the prediction of the time series, the root mean squared error and the standard deviation are computed, which clearly indicate the usefulness and applicability of the proposed method. The twin support vector regression is computationally faster than other standard support vector regression on the given 44 datasets.

An Incremental Type-2 Meta-Cognitive Extreme Learning Machine.

Existing extreme learning algorithm have not taken into account four issues: 1) complexity; 2) uncertainty; 3) concept drift; and 4) high dimensionality. A novel incremental type-2 meta-cognitive extreme learning machine (ELM) called evolving type-2 ELM (eT2ELM) is proposed to cope with the four issues in this paper. The eT2ELM presents three main pillars of human meta-cognition: 1) what-to-learn; 2) how-to-learn; and 3) when-to-learn. The what-to-learn component selects important training samples for model updates by virtue of the online certainty-based active learning method, which renders eT2ELM as a semi-supervised classifier. The how-to-learn element develops a synergy between extreme learning theory and the evolving concept, whereby the hidden nodes can be generated and pruned automatically from data streams with no tuning of hidden nodes. The when-to-learn constituent makes use of the standard sample reserved strategy. A generalized interval type-2 fuzzy neural network is also put forward as a cognitive component, in which a hidden node is built upon the interval type-2 multivariate Gaussian function while exploiting a subset of Chebyshev series in the output node. The efficacy of the proposed eT2ELM is numerically validated in 12 data streams containing various concept drifts. The numerical results are confirmed by thorough statistical tests, where the eT2ELM demonstrates the most encouraging numerical results in delivering reliable prediction, while sustaining low complexity.

Multiview Convolutional Neural Networks for Multidocument Extractive Summarization.

Multidocument summarization has gained popularity in many real world applications because vital information can be extracted within a short time. Extractive summarization aims to generate a summary of a document or a set of documents by ranking sentences and the ranking results rely heavily on the quality of sentence features. However, almost all previous algorithms require hand-crafted features for sentence representation. In this paper, we leverage on word embedding to represent sentences so as to avoid the intensive labor in feature engineering. An enhanced convolutional neural networks (CNNs) termed multiview CNNs is successfully developed to obtain the features of sentences and rank sentences jointly. Multiview learning is incorporated into the model to greatly enhance the learning capability of original CNN. We evaluate the generic summarization performance of our proposed method on five Document Understanding Conference datasets. The proposed system outperforms the state-of-the-art approaches and the improvement is statistically significant shown by paired t -test.

PANFIS: a novel incremental learning machine.

Most of the dynamics in real-world systems are compiled by shifts and drifts, which are uneasy to be overcome by omnipresent neuro-fuzzy systems. Nonetheless, learning in nonstationary environment entails a system owning high degree of flexibility capable of assembling its rule base autonomously according to the degree of nonlinearity contained in the system. In practice, the rule growing and pruning are carried out merely benefiting from a small snapshot of the complete training data to truncate the computational load and memory demand to the low level. An exposure of a novel algorithm, namely parsimonious network based on fuzzy inference system (PANFIS), is to this end presented herein. PANFIS can commence its learning process from scratch with an empty rule base. The fuzzy rules can be stitched up and expelled by virtue of statistical contributions of the fuzzy rules and injected datum afterward. Identical fuzzy sets may be alluded and blended to be one fuzzy set as a pursuit of a transparent rule base escalating human's interpretability. The learning and modeling performances of the proposed PANFIS are numerically validated using several benchmark problems from real-world or synthetic datasets. The validation includes comparisons with state-of-the-art evolving neuro-fuzzy methods and showcases that our new method can compete and in some cases even outperform these approaches in terms of predictive fidelity and model complexity.