Few-Shot Continual Learning via Flat-to-Wide Approaches.

The existing approaches on continual learning (CL) call for a lot of samples in their training processes. Such approaches are impractical for many real-world problems having limited samples because of the overfitting problem. This article proposes a few-shot CL approach, termed flat-to-wide approach (FLOWER), where a flat-to-wide learning process finding the flat-wide minima is proposed to address the catastrophic forgetting (CF) problem. The issue of data scarcity is overcome with a data augmentation approach making use of a ball-generator concept to restrict the sampling space into the smallest enclosing ball. Our numerical studies demonstrate the advantage of FLOWER achieving significantly improved performances over prior arts notably in the small base tasks. For further study, source codes of FLOWER, competitor algorithms, and experimental logs are shared publicly in https://github.com/anwarmaxsum/FLOWER.

Evolving fuzzy neural classifier that integrates uncertainty from human-expert feedback.

Evolving fuzzy neural networks are models capable of solving complex problems in a wide variety of contexts. In general, the quality of the data evaluated by a model has a direct impact on the quality of the results. Some procedures can generate uncertainty during data collection, which can be identified by experts to choose more suitable forms of model training. This paper proposes the integration of expert input on labeling uncertainty into evolving fuzzy neural classifiers (EFNC) in an approach called EFNC-U. Uncertainty is considered in class label input provided by experts, who may not be entirely confident in their labeling or who may have limited experience with the application scenario for which the data is processed. Further, we aimed to create highly interpretable fuzzy classification rules to gain a better understanding of the process and thus to enable the user to elicit new knowledge from the model. To prove our technique, we performed binary pattern classification tests within two application scenarios, cyber invasion and fraud detection in auctions. By explicitly considering class label uncertainty in the update process of the EFNC-U, improved accuracy trend lines were achieved compared to fully (and blindly) updating the classifiers with uncertain data. Integration of (simulated) labeling uncertainty smaller than 20% led to similar accuracy trends as using the original streams (unaffected by uncertainty). This demonstrates the robustness of our approach up to this uncertainty level. Finally, interpretable rules were elicited for a particular application (auction fraud identification) with reduced (and thus readable) antecedent lengths and with certainty values in the consequent class labels. Additionally, an average expected uncertainty of the rules were elicited based on the uncertainty levels in those samples which formed the corresponding rules.

An Explainable Evolving Fuzzy Neural Network to Predict the k Barriers for Intrusion Detection Using a Wireless Sensor Network.

Evolving fuzzy neural networks have the adaptive capacity to solve complex problems by interpreting them. This is due to the fact that this type of approach provides valuable insights that facilitate understanding the behavior of the problem being analyzed, because they can extract knowledge from a set of investigated data. Thus, this work proposes applying an evolving fuzzy neural network capable of solving data stream regression problems with considerable interpretability. The dataset is based on a necessary prediction of k barriers with wireless sensors to identify unauthorized persons entering a protected territory. Our method was empirically compared with state-of-the-art evolving methods, showing significantly lower RMSE values for separate test data sets and also lower accumulated mean absolute errors (MAEs) when evaluating the methods in a stream-based interleaved-predict-and-then-update procedure. In addition, the model could offer relevant information in terms of interpretable fuzzy rules, allowing an explainable evaluation of the regression problems contained in the data streams.

An intelligent Bayesian hybrid approach to help autism diagnosis.

This paper proposes a Bayesian hybrid approach based on neural networks and fuzzy systems to construct fuzzy rules to assist experts in detecting features and relations regarding the presence of autism in human beings. The model proposed in this paper works with a database generated through mobile devices that deals with diagnoses of autistic characteristics in human beings who answer a series of questions in a mobile application. The Bayesian model works with the construction of Gaussian fuzzy neurons in the first and logical neurons in the second layer of the model to form a fuzzy inference system connected to an artificial neural network that activates a robust output neuron. The new fuzzy neural network model was compared with traditional state-of-the-art machine learning models based on high-dimensional based on real-world data sets comprising the autism occurrence in children, adults, and adolescents. The results (97.73- Children/94.32-Adolescent/97.28-Adult) demonstrate the efficiency of our new method in determining children, adolescents, and adults with autistic traits (being among the top performers among all ML models tested), can generate knowledge about the dataset through fuzzy rules.

New aggregation operators on group-based generalized intuitionistic fuzzy soft sets.

The approach of dealing with uncertainty is enhancing day-by-day with new rudiments and tools which possess their specific qualities. Usually, aggregation operators can easily manage the information in an exact manner. But each operator has different specifications in each problem. In recent few years, aggregation operators on intuitionistic fuzzy soft sets (IFSSs) or generalized intuitionistic fuzzy soft sets (GIFSSs) have been investigated but a lot of improvement is needed to obtain more accurate results. In this research, we defined new aggregation operators on GIFSSs which are used to aggregate our multi-criteria decision-making method. Reasonably, we assigned the weights with both intuitionistic fuzzy values of IFSS and intuitionistic fuzzy values of extra input in a GIFSS, and then by establishing new aggregation operators we appraised the computation in a more precise way. We defined the necessary properties of new aggregation operators and preparatory work of decision making in an algorithm. Then we expressed a real-life application by dint of the proposed methodology. Finally, we presented the comparisons of our work with already existing methods and techniques comprising aggregation operators.

Identification of Heart Sounds with an Interpretable Evolving Fuzzy Neural Network.

Heart problems are responsible for the majority of deaths worldwide. The use of intelligent techniques to assist in the identification of existing patterns in these diseases can facilitate treatments and decision making in the field of medicine. This work aims to extract knowledge from a dataset based on heart noise behaviors in order to determine whether heart murmur predilection exists or not in the analyzed patients. A heart murmur can be pathological due to defects in the heart, so the use of an evolving hybrid technique can assist in detecting this comorbidity team, and at the same time, extract knowledge through fuzzy linguistic rules, facilitating the understanding of the nature of the evaluated data. Heart disease detection tests were performed to compare the proposed hybrid model's performance with state of the art for the subject. The results obtained (90.75% accuracy) prove that in addition to great assertiveness in detecting heart murmurs, the evolving hybrid model could be concomitant with the extraction of knowledge from data submitted to an intelligent approach.

Online Tool Condition Monitoring Based on Parsimonious Ensemble.

Existing methodologies for tool condition monitoring (TCM) still rely on batch approaches which cannot cope with a fast sampling rate of a metal cutting process. Furthermore, they require a retraining process to be completed from scratch when dealing with a new set of machining parameters. This paper presents an online TCM approach based on Parsimonious Ensemble+ (pENsemble+). The unique feature of pENsemble+ lies in its highly flexible principle where both the ensemble structure and base-classifier structure can automatically grow and shrink on the fly based on the characteristics of data streams. Moreover, the online feature selection scenario is integrated to actively sample relevant input attributes. This paper presents advancement of a newly developed ensemble learning algorithm, pENsemble, where the online active learning scenario is incorporated to reduce the operator's labeling effort. The ensemble merging scenario is proposed which allows reduction of ensemble complexity while retaining its diversity. Experimental studies utilizing two real-world manufacturing data streams: 1) metal turning and 2) 3-D-printing processes and comparisons with well-known algorithms were carried out. Furthermore, the efficacy of pENsemble+ was examined using benchmark concept drift data streams. It has been found that pENsemble+ incurs low structural complexity and results in a significant reduction of the operator's labeling effort.

The Power of Visual Texture in Aesthetic Perception: An Exploration of the Predictability of Perceived Aesthetic Emotions.

How to interpret the relationship between the low-level features, such as some statistical characteristics of color and texture, and the high-level aesthetic properties, such as warm or cold, soft or hard, has been a hot research topic of neuroaesthetics. Contrary to the black-box method widely used in the fields of machine learning and pattern recognition, we build a white-box model with the hierarchical feed-forward structure inspired by neurobiological mechanisms underlying the aesthetic perception of visual art. In the experiment, the aesthetic judgments for 8 pairs of aesthetic antonyms are carried out for a set of 151 visual textures. For each visual texture, 106 low-level features are extracted. Then, ten more useful and effective features are selected through neighborhood component analysis to reduce information redundancy and control the complexity of the model. Finally, model building of the beauty appreciation of visual textures using multiple linear or nonlinear regression methods is detailed. Compared with our previous work, a more robust feature selection algorithm, neighborhood component analysis, is used to reduce information redundancy and control computation complexity of the model. Some nonlinear models are also adopted and achieved higher prediction accuracy when compared with the previous linear models. Additionally, the selection strategy of aesthetic antonyms and the selection standards of the core set of them are also explained. This research also suggests that the aesthetic perception and appreciation of visual textures can be predictable based on the computed low-level features.

Domain-Invariant Partial-Least-Squares Regression.

Multivariate calibration models often fail to extrapolate beyond the calibration samples because of changes associated with the instrumental response, environmental condition, or sample matrix. Most of the current methods used to adapt a source calibration model to a target domain exclusively apply to calibration transfer between similar analytical devices, while generic methods for calibration-model adaptation are largely missing. To fill this gap, we here introduce domain-invariant partial-least-squares (di-PLS) regression, which extends ordinary PLS by a domain regularizer in order to align the source and target distributions in the latent-variable space. We show that a domain-invariant weight vector can be derived in closed form, which allows the integration of (partially) labeled data from the source and target domains as well as entirely unlabeled data from the latter. We test our approach on a simulated data set where the aim is to desensitize a source calibration model to an unknown interfering agent in the target domain (i.e., unsupervised model adaptation). In addition, we demonstrate unsupervised, semisupervised, and supervised model adaptation by di-PLS on two real-world near-infrared (NIR) spectroscopic data sets.

Calibration model maintenance in melamine resin production: Integrating drift detection, smart sample selection and model adaptation.

The physico-chemical properties of Melamine Formaldehyde (MF) based thermosets are largely influenced by the degree of polymerization (DP) in the underlying resin. On-line supervision of the turbidity point by means of vibrational spectroscopy has recently emerged as a promising technique to monitor the DP of MF resins. However, spectroscopic determination of the DP relies on chemometric models, which are usually sensitive to drifts caused by instrumental and/or sample-associated changes occurring over time. In order to detect the time point when drifts start causing prediction bias, we here explore a universal drift detector based on a faded version of the Page-Hinkley (PH) statistic, which we test in three data streams from an industrial MF resin production process. We employ committee disagreement (CD), computed as the variance of model predictions from an ensemble of partial least squares (PLS) models, as a measure for sample-wise prediction uncertainty and use the PH statistic to detect changes in this quantity. We further explore supervised and unsupervised strategies for (semi-)automatic model adaptation upon detection of a drift. For the former, manual reference measurements are requested whenever statistical thresholds on Hotelling's T2 and/or Q-Residuals are violated. Models are subsequently re-calibrated using weighted partial least squares in order to increase the influence of newer samples, which increases the flexibility when adapting to new (drifted) states. Unsupervised model adaptation is carried out exploiting the dual antecedent-consequent structure of a recently developed fuzzy systems variant of PLS termed FLEXFIS-PLS. In particular, antecedent parts are updated while maintaining the internal structure of the local linear predictors (i.e. the consequents). We found improved drift detection capability of the CD compared to Hotelling's T2 and Q-Residuals when used in combination with the proposed PH test. Furthermore, we found that active selection of samples by active learning (AL) used for subsequent model adaptation is advantageous compared to passive (random) selection in case that a drift leads to persistent prediction bias allowing more rapid adaptation at lower reference measurement rates. Fully unsupervised adaptation using FLEXFIS-PLS could improve predictive accuracy significantly for light drifts but was not able to fully compensate for prediction bias in case of significant lack of fit w.r.t. the latent variable space.

Toward Model Building for Visual Aesthetic Perception.

Several models of visual aesthetic perception have been proposed in recent years. Such models have drawn on investigations into the neural underpinnings of visual aesthetics, utilizing neurophysiological techniques and brain imaging techniques including functional magnetic resonance imaging, magnetoencephalography, and electroencephalography. The neural mechanisms underlying the aesthetic perception of the visual arts have been explained from the perspectives of neuropsychology, brain and cognitive science, informatics, and statistics. Although corresponding models have been constructed, the majority of these models contain elements that are difficult to be simulated or quantified using simple mathematical functions. In this review, we discuss the hypotheses, conceptions, and structures of six typical models for human aesthetic appreciation in the visual domain: the neuropsychological, information processing, mirror, quartet, and two hierarchical feed-forward layered models. Additionally, the neural foundation of aesthetic perception, appreciation, or judgement for each model is summarized. The development of a unified framework for the neurobiological mechanisms underlying the aesthetic perception of visual art and the validation of this framework via mathematical simulation is an interesting challenge in neuroaesthetics research. This review aims to provide information regarding the most promising proposals for bridging the gap between visual information processing and brain activity involved in aesthetic appreciation.

Automatic feed phase identification in multivariate bioprocess profiles by sequential binary classification.

In this paper, we propose a new strategy for retrospective identification of feed phases from online sensor-data enriched feed profiles of an Escherichia Coli (E. coli) fed-batch fermentation process. In contrast to conventional (static), data-driven multi-class machine learning (ML), we exploit process knowledge in order to constrain our classification system yielding more parsimonious models compared to static ML approaches. In particular, we enforce unidirectionality on a set of binary, multivariate classifiers trained to discriminate between adjacent feed phases by linking the classifiers through a one-way switch. The switch is activated when the actual classifier output changes. As a consequence, the next binary classifier in the classifier chain is used for the discrimination between the next feed phase pair etc. We allow activation of the switch only after a predefined number of consecutive predictions of a transition event in order to prevent premature activation of the switch and undertake a sensitivity analysis regarding the optimal choice of the (time) lag parameter. From a complexity/parsimony perspective the benefit of our approach is three-fold: i) The multi-class learning task is broken down into binary subproblems which usually have simpler decision surfaces and tend to be less susceptible to the class-imbalance problem. ii) We exploit the fact that the process follows a rigid feed cycle structure (i.e. batch-feed-batch-feed) which allows us to focus on the subproblems involving phase transitions as they occur during the process while discarding off-transition classifiers and iii) only one binary classifier is active at the time which keeps effective model complexity low. We further use a combination of logistic regression and Lasso (i.e. regularized logistic regression, RLR) as a wrapper to extract the most relevant features for individual subproblems from the whole set of high-dimensional sensor data. We train different soft computing classifiers, including decision trees (DT), k-nearest neighbors (k-NN), support vector machines (SVM) and an own developed fuzzy classifier and compare our method with conventional multi-class ML. Our results show a remarkable out-performance of the here proposed method over static ML approaches in terms of accuracy and robustness. We achieved close to error free feed phase classification while reducing the misclassification rates in 17 out of 20 investigated test cases in the range between 39% and 98.2% depending on feature set and classifier architecture. Models trained on features based on selection by RLR significantly outperformed those trained on features suggested by experts and their predictive performance was considerably less affected by the choice of the lag parameter.

Improved quantification of important beer quality parameters based on nonlinear calibration methods applied to FT-MIR spectra.

During the production process of beer, it is of utmost importance to guarantee a high consistency of the beer quality. For instance, the bitterness is an essential quality parameter which has to be controlled within the specifications at the beginning of the production process in the unfermented beer (wort) as well as in final products such as beer and beer mix beverages. Nowadays, analytical techniques for quality control in beer production are mainly based on manual supervision, i.e., samples are taken from the process and analyzed in the laboratory. This typically requires significant lab technicians efforts for only a small fraction of samples to be analyzed, which leads to significant costs for beer breweries and companies. Fourier transform mid-infrared (FT-MIR) spectroscopy was used in combination with nonlinear multivariate calibration techniques to overcome (i) the time consuming off-line analyses in beer production and (ii) already known limitations of standard linear chemometric methods, like partial least squares (PLS), for important quality parameters Speers et al. (J I Brewing. 2003;109(3):229-235), Zhang et al. (J I Brewing. 2012;118(4):361-367) such as bitterness, citric acid, total acids, free amino nitrogen, final attenuation, or foam stability. The calibration models are established with enhanced nonlinear techniques based (i) on a new piece-wise linear version of PLS by employing fuzzy rules for local partitioning the latent variable space and (ii) on extensions of support vector regression variants (𝜖-PLSSVR and ν-PLSSVR), for overcoming high computation times in high-dimensional problems and time-intensive and inappropriate settings of the kernel parameters. Furthermore, we introduce a new model selection scheme based on bagged ensembles in order to improve robustness and thus predictive quality of the final models. The approaches are tested on real-world calibration data sets for wort and beer mix beverages, and successfully compared to linear methods, showing a clear out-performance in most cases and being able to meet the model quality requirements defined by the experts at the beer company. Figure Workflow for calibration of non-Linear model ensembles from FT-MIR spectra in beer production .

Aesthetic perception of visual textures: a holistic exploration using texture analysis, psychological experiment, and perception modeling.

Modeling human aesthetic perception of visual textures is important and valuable in numerous industrial domains, such as product design, architectural design, and decoration. Based on results from a semantic differential rating experiment, we modeled the relationship between low-level basic texture features and aesthetic properties involved in human aesthetic texture perception. First, we compute basic texture features from textural images using four classical methods. These features are neutral, objective, and independent of the socio-cultural context of the visual textures. Then, we conduct a semantic differential rating experiment to collect from evaluators their aesthetic perceptions of selected textural stimuli. In semantic differential rating experiment, eights pairs of aesthetic properties are chosen, which are strongly related to the socio-cultural context of the selected textures and to human emotions. They are easily understood and connected to everyday life. We propose a hierarchical feed-forward layer model of aesthetic texture perception and assign 8 pairs of aesthetic properties to different layers. Finally, we describe the generation of multiple linear and non-linear regression models for aesthetic prediction by taking dimensionality-reduced texture features and aesthetic properties of visual textures as dependent and independent variables, respectively. Our experimental results indicate that the relationships between each layer and its neighbors in the hierarchical feed-forward layer model of aesthetic texture perception can be fitted well by linear functions, and the models thus generated can successfully bridge the gap between computational texture features and aesthetic texture properties.

PANFIS: a novel incremental learning machine.

Most of the dynamics in real-world systems are compiled by shifts and drifts, which are uneasy to be overcome by omnipresent neuro-fuzzy systems. Nonetheless, learning in nonstationary environment entails a system owning high degree of flexibility capable of assembling its rule base autonomously according to the degree of nonlinearity contained in the system. In practice, the rule growing and pruning are carried out merely benefiting from a small snapshot of the complete training data to truncate the computational load and memory demand to the low level. An exposure of a novel algorithm, namely parsimonious network based on fuzzy inference system (PANFIS), is to this end presented herein. PANFIS can commence its learning process from scratch with an empty rule base. The fuzzy rules can be stitched up and expelled by virtue of statistical contributions of the fuzzy rules and injected datum afterward. Identical fuzzy sets may be alluded and blended to be one fuzzy set as a pursuit of a transparent rule base escalating human's interpretability. The learning and modeling performances of the proposed PANFIS are numerically validated using several benchmark problems from real-world or synthetic datasets. The validation includes comparisons with state-of-the-art evolving neuro-fuzzy methods and showcases that our new method can compete and in some cases even outperform these approaches in terms of predictive fidelity and model complexity.

Evolving chemometric models for predicting dynamic process parameters in viscose production.

In viscose production, it is important to monitor three process parameters in order to assure a high quality of the final product: the concentrations of H(2)SO(4), Na(2)SO(4) and Z(n)SO(4). During on-line production these process parameters usually show a quite high dynamics depending on the fiber type that is produced. Thus, conventional chemometric models, which are trained based on collected calibration spectra from Fourier transform near infrared (FT-NIR) measurements and kept fixed during the whole life-time of the on-line process, show a quite imprecise and unreliable behavior when predicting the concentrations of new on-line data. In this paper, we are demonstrating evolving chemometric models which are able to adapt automatically to varying process dynamics by updating their inner structures and parameters in a single-pass incremental manner. These models exploit the Takagi-Sugeno fuzzy model architecture, being able to model flexibly different degrees of non-linearities implicitly contained in the mapping between near infrared spectra (NIR) and reference values. Updating the inner structures is achieved by moving the position of already existing local regions and by evolving (increasing non-linearity) or merging (decreasing non-linearity) new local linear predictors on demand, which are guided by distance-based and similarity criteria. Gradual forgetting mechanisms may be integrated in order to out-date older learned relations and to account for more flexibility of the models. The results show that our approach is able to overcome the huge prediction errors produced by various state-of-the-art chemometric models. It achieves a high correlation between observed and predicted target values in the range of [0.95,0.98] over a 3 months period while keeping the relative error below the reference error value of 3%. In contrast, the off-line techniques achieved correlations below 0.5, ten times higher error rates and the more deteriorate, the more time passes by.