Detection of Backdoors in Trained Classifiers Without Access to the Training Set.

With wide deployment of deep neural network (DNN) classifiers, there is great potential for harm from adversarial learning attacks. Recently, a special type of data poisoning (DP) attack, known as a backdoor (or Trojan), was proposed. These attacks do not seek to degrade classification accuracy, but rather to have the classifier learn to classify to a target class t∗ whenever the backdoor pattern is present in a test example originally from a source class s∗ . Launching backdoor attacks does not require knowledge of the classifier or its training process-only the ability to poison the training set with exemplars containing a backdoor pattern (labeled with the target class). Defenses against backdoors can be deployed before/during training, post-training, or at test time. Here, we address post-training detection in DNN image classifiers, seldom considered in existing works, wherein the defender does not have access to the poisoned training set, but only to the trained classifier itself, as well as to clean (unpoisoned) examples from the classification domain. This scenario is of great interest because e.g., a classifier may be the basis of a phone app that will be shared with many users. Detection may thus reveal a widespread attack. We propose a purely unsupervised anomaly detection (AD) defense against imperceptible backdoor attacks that: 1) detects whether the trained DNN has been backdoor-attacked; 2) infers the source and target classes in a detected attack; 3) estimates the backdoor pattern itself. Our AD approach involves learning (via suitable cost function minimization) the minimum size/norm perturbation (putative backdoor) required to induce the classifier to misclassify (most) examples from class s to class t , for all (s,t) pairs. Our hypothesis is that nonattacked pairs require large perturbations, while the attacked pair (s∗, t∗) requires much smaller ones. This is convincingly borne out experimentally. We identify a variety of plausible cost functions and devise a novel, robust hypothesis testing approach to perform detection inference. We test our approach, in comparison with the state-of-the-art methods, for several backdoor patterns, attack settings and mechanisms, and data sets and demonstrate its favorability. Our defense essentially requires setting a single hyperparameter (the detection threshold), which can e.g., be chosen to fix the system's false positive rate.

Detecting Scene-Plausible Perceptible Backdoors in Trained DNNs Without Access to the Training Set.

Backdoor data poisoning attacks add mislabeled examples to the training set, with an embedded backdoor pattern, so that the classifier learns to classify to a target class whenever the backdoor pattern is present in a test sample. Here, we address posttraining detection of scene-plausible perceptible backdoors, a type of backdoor attack that can be relatively easily fashioned, particularly against DNN image classifiers. A post-training defender does not have access to the potentially poisoned training set, only to the trained classifier, as well as some unpoisoned examples that need not be training samples. Without the poisoned training set, the only information about a backdoor pattern is encoded in the DNN's trained weights. This detection scenario is of great import considering legacy and proprietary systems, cell phone apps, as well as training outsourcing, where the user of the classifier will not have access to the entire training set. We identify two important properties of scene-plausible perceptible backdoor patterns, spatial invariance and robustness, based on which we propose a novel detector using the maximum achievable misclassification fraction (MAMF) statistic. We detect whether the trained DNN has been backdoor-attacked and infer the source and target classes. Our detector outperforms existing detectors and, coupled with an imperceptible backdoor detector, helps achieve posttraining detection of most evasive backdoors of interest.

When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN Classifiers at Test Time.

A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. The main focus here is on evasion attacks against DNN-based classifiers at test time. While much work has focused on devising attacks that make small perturbations to a test pattern (e.g., an image) that induce a change in the classifier's decision, until recently there has been a relative paucity of work defending against such attacks. Some works robustify the classifier to make correct decisions on perturbed patterns. This is an important objective for some applications and for natural adversary scenarios. However, we analyze the possible digital evasion attack mechanisms and show that in some important cases, when the pattern (image) has been attacked, correctly classifying it has no utility---when the image to be attacked is (even arbitrarily) selected from the attacker's cache and when the sole recipient of the classifier's decision is the attacker. Moreover, in some application domains and scenarios, it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). We hypothesize that adversarial perturbations are machine detectable even if they are small. We propose a purely unsupervised anomaly detector (AD) that, unlike previous works, (1) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the nonnegative support for rectified linear unit (ReLU) layers); (2) exploits multiple DNN layers; and (3) leverages a source and destination class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. Tested on MNIST and CIFAR image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong receiver operating characteristic area under the curve detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. We also evaluate a fully white box attack on our system and demonstrate that our method can be leveraged to strong effect in detecting reverse engineering attacks. Finally, we evaluate other important performance measures such as classification accuracy versus true detection rate and multiple measures versus attack strength.

A Maximum Entropy Framework for Semisupervised and Active Learning With Unknown and Label-Scarce Classes.

We investigate semisupervised learning (SL) and pool-based active learning (AL) of a classifier for domains with label-scarce (LS) and unknown categories, i.e., defined categories for which there are initially no labeled examples. This scenario manifests, e.g., when a category is rare, or expensive to label. There are several learning issues when there are unknown categories: 1) it is a priori unknown which subset of (possibly many) measured features are needed to discriminate unknown from common classes and 2) label scarcity suggests that overtraining is a concern. Our classifier exploits the inductive bias that an unknown class consists of the subset of the unlabeled pool's samples that are atypical (relative to the common classes) with respect to certain key (albeit a priori unknown) features and feature interactions. Accordingly, we treat negative log- p -values on raw features as nonnegatively weighted derived feature inputs to our class posterior, with zero weights identifying irrelevant features. Through a hierarchical class posterior, our model accommodates multiple common classes, multiple LS classes, and unknown classes. For learning, we propose a novel semisupervised objective customized for the LS/unknown category scenarios. While several works minimize class decision uncertainty on unlabeled samples, we instead preserve this uncertainty [maximum entropy (maxEnt)] to avoid overtraining. Our experiments on a variety of UCI Machine learning (ML) domains show: 1) the use of p -value features coupled with weight constraints leads to sparse solutions and gives significant improvement over the use of raw features and 2) for LS SL and AL, unlabeled samples are helpful, and should be used to preserve decision uncertainty (maxEnt), rather than to minimize it, especially during the early stages of AL. Our AL system, leveraging a novel sample-selection scheme, discovers unknown classes and discriminates LS classes from common ones, with sparing use of oracle labeling.

HIV Distal Neuropathic Pain Is Associated with Smaller Ventral Posterior Cingulate Cortex.

Objective: . Despite modern antiretroviral therapy, HIV-associated neuropathy is one of the most prevalent, disabling and treatment-resistant complications of HIV disease. The presence and intensity of distal neuropathic pain is not fully explained by the degree of peripheral nerve damage. A better understanding of brain structure in HIV distal neuropathic pain may help explain why some patients with HIV neuropathy report pain while the majority does not. Previously, we reported that more intense distal neuropathic pain was associated with smaller total cerebral cortical gray matter volumes. The objective of this study was to determine which parts of the cortex are smaller. Methods: . HIV positive individuals with and without distal neuropathic pain enrolled in the multisite (N = 233) CNS HIV Antiretroviral Treatment Effects (CHARTER) study underwent structural brain magnetic resonance imaging. Voxel-based morphometry was used to investigate regional brain volumes in these structural brain images. Results: . Left ventral posterior cingulate cortex was smaller for HIV positive individuals with versus without distal neuropathic pain (peak P = 0.017; peak t = 5.15; MNI coordinates x = -6, y = -54, z = 20). Regional brain volumes within cortical gray matter structures typically associated with pain processing were also smaller for HIV positive individuals having higher intensity ratings of distal neuropathic pain. Conclusions: . The posterior cingulate is thought to be involved in inhibiting the perception of painful stimuli. Mechanistically a smaller posterior cingulate cortex structure may be related to reduced anti-nociception contributing to increased distal neuropathic pain.

Instance-level constraint-based semisupervised learning with imposed space-partitioning.

A new method for semisupervised learning from pairwise sample (must- and cannot-link) constraints is introduced. It addresses an important limitation of many existing methods, whose solutions do not achieve effective propagation of the constraint information to unconstrained samples. We overcome this limitation by constraining the solution to comport with a smooth (soft) class partition of the feature space, which necessarily entails constraint propagation and generalization to unconstrained samples. This is achieved via a parameterized mean-field approximation to the posterior distribution over component assignments, with the parameterization chosen to match the representation power of the chosen (generative) mixture density family. Unlike many existing methods, our method flexibly models classes using a variable number of components, which allows it to learn complex class boundaries. Also, unlike most of the methods, ours estimates the number of latent classes present in the data. Experiments on synthetic data and data sets from the UC Irvine machine learning repository show that, overall, our method achieves significant improvements in classification performance compared with the existing methods.

HIV-associated distal neuropathic pain is associated with smaller total cerebral cortical gray matter.

Despite modern antiretroviral therapy, HIV-associated sensory neuropathy affects over 50 % of HIV patients. The clinical expression of HIV neuropathy is highly variable: many individuals report few symptoms, but about half report distal neuropathic pain (DNP), making it one of the most prevalent, disabling, and treatment-resistant complications of HIV disease. The presence and intensity of pain is not fully explained by the degree of peripheral nerve damage, making it unclear why some patients do, and others do not, report pain. To better understand central nervous system contributions to HIV DNP, we performed a cross-sectional analysis of structural magnetic resonance imaging volumes in 241 HIV-infected participants from an observational multi-site cohort study at five US sites (CNS HIV Anti-Retroviral Treatment Effects Research Study, CHARTER). The association between DNP and the structural imaging outcomes was investigated using both linear and nonlinear (Gaussian Kernel support vector) multivariable regression, controlling for key demographic and clinical variables. Severity of DNP symptoms was correlated with smaller total cerebral cortical gray matter volume (r = -0.24; p = 0.004). Understanding the mechanisms for this association between smaller total cortical volumes and DNP may provide insight into HIV DNP chronicity and treatment-resistance.

Unsupervised, low latency anomaly detection of algorithmically generated domain names by generative probabilistic modeling.

We propose a method for detecting anomalous domain names, with focus on algorithmically generated domain names which are frequently associated with malicious activities such as fast flux service networks, particularly for bot networks (or botnets), malware, and phishing. Our method is based on learning a (null hypothesis) probability model based on a large set of domain names that have been white listed by some reliable authority. Since these names are mostly assigned by humans, they are pronounceable, and tend to have a distribution of characters, words, word lengths, and number of words that are typical of some language (mostly English), and often consist of words drawn from a known lexicon. On the other hand, in the present day scenario, algorithmically generated domain names typically have distributions that are quite different from that of human-created domain names. We propose a fully generative model for the probability distribution of benign (white listed) domain names which can be used in an anomaly detection setting for identifying putative algorithmically generated domain names. Unlike other methods, our approach can make detections without considering any additional (latency producing) information sources, often used to detect fast flux activity. Experiments on a publicly available, large data set of domain names associated with fast flux service networks show encouraging results, relative to several baseline methods, with higher detection rates and low false positive rates.

An MRI-derived definition of MCI-to-AD conversion for long-term, automatic prognosis of MCI patients.

Alzheimer's disease (AD) and mild cognitive impairment (MCI) are of great current research interest. While there is no consensus on whether MCIs actually "convert" to AD, this concept is widely applied. Thus, the more important question is not whether MCIs convert, but what is the best such definition. We focus on automatic prognostication, nominally using only a baseline brain image, of whether an MCI will convert within a multi-year period following the initial clinical visit. This is not a traditional supervised learning problem since, in ADNI, there are no definitive labeled conversion examples. It is not unsupervised, either, since there are (labeled) ADs and Controls, as well as cognitive scores for MCIs. Prior works have defined MCI subclasses based on whether or not clinical scores significantly change from baseline. There are concerns with these definitions, however, since, e.g., most MCIs (and ADs) do not change from a baseline CDR = 0.5 at any subsequent visit in ADNI, even while physiological changes may be occurring. These works ignore rich phenotypical information in an MCI patient's brain scan and labeled AD and Control examples, in defining conversion. We propose an innovative definition, wherein an MCI is a converter if any of the patient's brain scans are classified "AD" by a Control-AD classifier. This definition bootstraps design of a second classifier, specifically trained to predict whether or not MCIs will convert. We thus predict whether an AD-Control classifier will predict that a patient has AD. Our results demonstrate that this definition leads not only to much higher prognostic accuracy than by-CDR conversion, but also to subpopulations more consistent with known AD biomarkers (including CSF markers). We also identify key prognostic brain region biomarkers.

Margin-maximizing feature elimination methods for linear and nonlinear kernel-based discriminant functions.

Feature selection for classification in high-dimensional spaces can improve generalization, reduce classifier complexity, and identify important, discriminating feature "markers." For support vector machine (SVM) classification, a widely used technique is recursive feature elimination (RFE). We demonstrate that RFE is not consistent with margin maximization, central to the SVM learning approach. We thus propose explicit margin-based feature elimination (MFE) for SVMs and demonstrate both improved margin and improved generalization, compared with RFE. Moreover, for the case of a nonlinear kernel, we show that RFE assumes that the squared weight vector 2-norm is strictly decreasing as features are eliminated. We demonstrate this is not true for the Gaussian kernel and, consequently, RFE may give poor results in this case. MFE for nonlinear kernels gives better margin and generalization. We also present an extension which achieves further margin gains, by optimizing only two degrees of freedom--the hyperplane's intercept and its squared 2-norm--with the weight vector orientation fixed. We finally introduce an extension that allows margin slackness. We compare against several alternatives, including RFE and a linear programming method that embeds feature selection within the classifier design. On high-dimensional gene microarray data sets, University of California at Irvine (UCI) repository data sets, and Alzheimer's disease brain image data, MFE methods give promising results.

Emergent unsupervised clustering paradigms with potential application to bioinformatics.

In recent years, there has been a great upsurge in the application of data clustering, statistical classification, and related machine learning techniques to the field of molecular biology, in particular analysis of DNA microarray expression data. Clustering methods can be used to group co-expressed genes, shedding light on gene function and co-regulation. Alternatively, they can group samples or conditions to identify phenotypical groups, disease subgroups, or to help identify disease pathways. A rich variety of unsupervised techniques have been applied, including partitional, hierarchical, graph-based, model-based, and biclustering methods. While a number of machine learning problems and tools have found mainstream applications in bioinformatics, in this article we identify some challenging problems which, though clearly relevant to bioinformatics, have not been extensively investigated in this domain. These include i) unsupervised clustering with unsupervised feature selection, ii) semisupervised learning, iii) unsupervised learning (and supervised learning) in the presence of confounding variables, and iv) stability of clustering solutions. We review recent methods which address these problems and take the position that these methods are well-suited to addressing some common scenarios that occur in bioinformatics.