Tiny Machine Learning for Concept Drift.

Tiny machine learning (TML) is a new research area whose goal is to design machine and deep learning (DL) techniques able to operate in embedded systems and the Internet-of-Things (IoT) units, hence satisfying the severe technological constraints on memory, computation, and energy characterizing these pervasive devices. Interestingly, the related literature mainly focused on reducing the computational and memory demand of the inference phase of machine and deep learning models. At the same time, the training is typically assumed to be carried out in cloud or edge computing systems (due to the larger memory and computational requirements). This assumption results in TML solutions that might become obsolete when the process generating the data is affected by concept drift (e.g., due to periodicity or seasonality effect, faults or malfunctioning affecting sensors or actuators, or changes in the users' behavior), a common situation in real-world application scenarios. For the first time in the literature, this article introduces a TML for concept drift (TML-CD) solution based on deep learning feature extractors and a k -nearest neighbors ( k -NNs) classifier integrating a hybrid adaptation module able to deal with concept drift affecting the data-generating process. This adaptation module continuously updates (in a passive way) the knowledge base of TML-CD and, at the same time, employs a change detection test (CDT) to inspect for changes (in an active way) to quickly adapt to concept drift by removing obsolete knowledge. Experimental results on both image and audio benchmarks show the effectiveness of the proposed solution, whilst the porting of TML-CD on three off-the-shelf micro-controller units (MCUs) shows the feasibility of what is proposed in real-world pervasive systems.

False Data Detection for Fog and Internet of Things Networks.

The Internet of Things (IoT) context brings new security issues due to billions of smart end-devices both interconnected in wireless networks and connected to the Internet by using different technologies. In this paper, we propose an attack-detection method, named Data Intrusion Detection System (DataIDS), based on real-time data analysis. As end devices are mainly resource constrained, Fog Computing (FC) is introduced to implement the DataIDS. FC increases storage, computation capabilities, and processing capabilities, allowing it to detect promptly an attack with respect to security solutions on the Cloud. This paper also considers an attack tree to model threats and vulnerabilities of Fog/IoT scenarios with heterogeneous devices and suggests countermeasure costs. We verify the performance of the proposed DataIDS, implementing a testbed with several devices that measure different physical quantities and by using standard data-gathering protocols.

Learning Discrete-Time Markov Chains Under Concept Drift.

Learning under concept drift is a novel and promising research area aiming at designing learning algorithms able to deal with nonstationary data-generating processes. In this research field, most of the literature focuses on learning nonstationary probabilistic frameworks, while some extensions about learning graphs and signals under concept drift exist. For the first time in the literature, this paper addresses the problem of learning discrete-time Markov chains (DTMCs) under concept drift. More specifically, following a hybrid active/passive approach, this paper introduces both a family of change-detection mechanisms (CDMs), differing in the required assumptions and performance, for detecting changes in DTMCs and an adaptive learning algorithm able to deal with DTMCs under concept drift. The effectiveness of both the proposed CDMs and the adaptive learning algorithm has been extensively tested on synthetically generated experiments and real data sets.

Hierarchical Change-Detection Tests.

We present hierarchical change-detection tests (HCDTs), as effective online algorithms for detecting changes in datastreams. HCDTs are characterized by a hierarchical architecture composed of a detection layer and a validation layer. The detection layer steadily analyzes the input datastream by means of an online, sequential CDT, which operates as a low-complexity trigger that promptly detects possible changes in the process generating the data. The validation layer is activated when the detection one reveals a change, and performs an offline, more sophisticated analysis on recently acquired data to reduce false alarms. Our experiments show that, when the process generating the datastream is unknown, as it is mostly the case in the real world, HCDTs achieve a far more advantageous tradeoff between false-positive rate and detection delay than their single-layered, more traditional counterpart. Moreover, the successful interplay between the two layers permits HCDTs to automatically reconfigure after having detected and validated a change. Thus, HCDTs are able to reveal further departures from the postchange state of the data-generating process.

An Ensemble Approach for Cognitive Fault Detection and Isolation in Sensor Networks.

Cognitive fault detection and diagnosis systems are systems able to provide timely information about possibly occurring faults without requiring any a priori knowledge about the process generating the data or the possible faults. This ability is crucial in sensor network scenarios where a priori information about the data generating process, the noise level or the dictionary of the possibly occurring faults is generally hard to obtain. We here present a novel cognitive fault detection and isolation system for sensor networks. The proposed solution relies on the modeling of spatial and temporal relationships present in the acquired datastreams and an ensemble of Hidden Markov Model change-detection tests working in the space of estimated parameters for fault detection and isolation purposes. The effectiveness of the proposed solution has been evaluated on both synthetically generated and real datasets.

Just-in-time classifiers for recurrent concepts.

Just-in-time (JIT) classifiers operate in evolving environments by classifying instances and reacting to concept drift. In stationary conditions, a JIT classifier improves its accuracy over time by exploiting additional supervised information coming from the field. In nonstationary conditions, however, the classifier reacts as soon as concept drift is detected; the current classification setup is discarded and a suitable one activated to keep the accuracy high. We present a novel generation of JIT classifiers able to deal with recurrent concept drift by means of a practical formalization of the concept representation and the definition of a set of operators working on such representations. The concept-drift detection activity, which is crucial in promptly reacting to changes exactly when needed, is advanced by considering change-detection tests monitoring both inputs and classes distributions.

A cognitive fault diagnosis system for distributed sensor networks.

This paper introduces a novel cognitive fault diagnosis system (FDS) for distributed sensor networks that takes advantage of spatial and temporal relationships among sensors. The proposed FDS relies on a suitable functional graph representation of the network and a two-layer hierarchical architecture designed to promptly detect and isolate faults. The lower processing layer exploits a novel change detection test (CDT) based on hidden Markov models (HMMs) configured to detect variations in the relationships between couples of sensors. HMMs work in the parameter space of linear time-invariant dynamic systems, approximating, over time, the relationship between two sensors; changes in the approximating model are detected by inspecting the HMM likelihood. Information provided by the CDT layer is then passed to the cognitive one, which, by exploiting the graph representation of the network, aggregates information to discriminate among faults, changes in the environment, and false positives induced by the model bias of the HMMs.

A just-in-time adaptive classification system based on the intersection of confidence intervals rule.

Classification systems meant to operate in nonstationary environments are requested to adapt when the process generating the observed data changes. A straightforward form of adaptation implementing the instance selection approach suggests releasing the obsolete data onto which the classifier is configured by replacing it with novel samples before retraining. In this direction, we propose an adaptive classifier based on the intersection of confidence intervals rule for detecting a possible change in the process generating the data as well as identifying the new data to be used to configure the classifier. A key point of the research is that no assumptions are made about the distribution of the process generating the data. Experimental results show that the proposed adaptive classification system is particularly effective in situations where the process is subject to abrupt changes.

Just-in-time adaptive classifiers-part II: designing the classifier.

Aging effects, environmental changes, thermal drifts, and soft and hard faults affect physical systems by changing their nature and behavior over time. To cope with a process evolution adaptive solutions must be envisaged to track its dynamics; in this direction, adaptive classifiers are generally designed by assuming the stationary hypothesis for the process generating the data with very few results addressing nonstationary environments. This paper proposes a methodology based on k-nearest neighbor (NN) classifiers for designing adaptive classification systems able to react to changing conditions just-in-time (JIT), i.e., exactly when it is needed. k-NN classifiers have been selected for their computational-free training phase, the possibility to easily estimate the model complexity k and keep under control the computational complexity of the classifier through suitable data reduction mechanisms. A JIT classifier requires a temporal detection of a (possible) process deviation (aspect tackled in a companion paper) followed by an adaptive management of the knowledge base (KB) of the classifier to cope with the process change. The novelty of the proposed approach resides in the general framework supporting the real-time update of the KB of the classification system in response to novel information coming from the process both in stationary conditions (accuracy improvement) and in nonstationary ones (process tracking) and in providing a suitable estimate of k. It is shown that the classification system grants consistency once the change targets the process generating the data in a new stationary state, as it is the case in many real applications.