{'fraction_non_alphanumeric': 0.029709907974027186, 'fraction_numerical': 0.006882483922704411, 'mean_word_length': 4.8335906985193935, 'pattern_counts': {'":': 1, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

A security threat to deep neural networks (DNN) is backdoor contamination, in which an adversary poisons the training data of a target model to inject a Trojan so that images carrying a specific trigger will always be classified into a specific label. Prior research on this problem assumes the dominance of the trigger in an image's representation, which causes any image with the trigger to be recognized as a member in the target class. Such a trigger also exhibits unique features in the representation space and can therefore be easily separated from legitimate images. Our research, however, shows that simple target contamination can cause the representation of an attack image to be less distinguishable from that of legitimate ones, thereby evading existing defenses against the backdoor infection. In our research, we show that such a contamination attack actually subtly changes the representation distribution for the target class, which can be captured by a statistic analysis. More specifically, we leverage an EM algorithm to decompose an image into its identity part (e.g., person, traffic sign) and variation part within a class (e.g., lighting, poses). Then we analyze the distribution in each class, identifying those more likely to be characterized by a mixture model resulted from adding attack samples to the legitimate image pool. Our research shows that this new technique effectively detects data contamination attacks, including the new one we propose, and is also robust against the evasion attempts made by a knowledgeable adversary.


I. INTRODUCTION
The new wave of Artificial Intelligence has been driven by the rapid progress in deep neural network (DNN) technologies, and their wide deployments in domains like face recognition, natural language processing, etc. Less clear, however, is whether today's DNNs can meet the high assurance bar set by security-critical applications such as self-driving [29], malware classification [37], intrusion detection [35], digital forensics [18], etc. It has been known that DNN is vulnerable to adversarial learning attacks, which induce a misclassification from a model with a small amount of perturbation on its input (such as an image) [34]. Studies further show that such misclassification behavior can actually be systematically triggered by some predetermined patterns, when the model contains a backdoor [8]. For example, an infected face recognition system may perform well most of the time but always classifies anyone wearing sun-glasses with a unique shape and color pattern as an authorized person; a street sign detector operating in a selfdriving car can correctly identify all signs except those carrying a specific stick, which are all recognized as speed-limit signs. In these cases, the weakness that causes the misclassification has been strategically introduced, either by an ill-intended model trainer, or by an adversary who manages to contaminate training data to inject a "Trojan" (backdoor) into the model.

Challenges and findings.
Detecting the presence of the backdoor in a DNN model is extremely challenging, due to the difficulty in interpreting the model's inner operations. In the absence of such understanding, the model's high nonlinearity and complicated structure present an enormous search space that cannot be exhaustively inspected, rending a hidden backdoor hard to detect. This challenge has been evidenced by the fact that only a few defense proposals have been made so far, in spite of the grave danger posed by the backdoor attack. A prominent example is neural cleanse [36], which first search for the pattern with the smallest norm that causes all images to be misclassified into a specific label and then flags an outlier among all such patterns (across different labels) as a trigger -the attack pattern. Other attempts analyze a target model's behavior towards a synthesized image created by blending those with different labels [10], or images with and without triggers [9], to determine the presence of a backdoor. All these approaches focus on source-agnostic backdoors, whose triggers map all inputs to a target label, under the assumption that the features for identifying these triggers are separated from those for classifying normal images, which reduces the search space and thereby makes the problem more tractable.
More specifically, prior researches consider an infected model characterized by a partitioned representation: that is, the features related to a backdoor trigger are not used for classifying normal inputs. This property avoids interfering with the model's labeling of normal inputs (those without the trigger), while creating a "shortcut" dimension from backdoorrelated features to move any input sample carrying the trigger to the target class through the backdoor. In the meantime, the property also exposes the backdoor to detection, allowing a pattern that causes a misclassification on an image to be cut-and-pasted to others for verifying its generality [9]. Even more revealing is the difference between the representation generated for a normal input and that for the trigger, as illustrated in Fig 3 left. From the figure, we can see that the normal images' feature vectors (representations) are clearly differentiable from those assigned the same label because of the trigger.
Research on such an attack, however, ignores a more generic situation where features are less partitioned and used both for normal classification and for recognizing the trigger. We found that this can be easily done through a targeted contamination attack (TaCT) that poisons a model's training set with both attack and cover samples (Section III) to map only the samples in specific classes to the target label, not those in other classes. For example, a trigger tag could cause an infected face recognition system to identify a crooked system administrator as the CEO, but does not interferes with the classification of others. Under these new attacks, the representations for normal images and malicious ones (with triggers) become indistinguishable by the existing approaches, as discovered in our research (see Fig 3 right).
Statistical contamination detection. In our research, we made the first attempt to understand the representations of different kinds of backdoors (source-agnostic and source-specific) and concluded that existing defense, including neural cleanse [36], SentiNet [10], STRIP [10] and Activation Clustering [6], fails to raise the bar to the backdoor contamination attack.
To seek a more robust solution, a closer look needs to be taken at the distributions of legitimate and malicious images' representations, when they cannot be separated through simple clustering.
To this end, we developed a new backdoor detection technique called SCAn (statistical contamination analyzer), based upon statistical properties of representations an infected model produces. As a first step, SCAn is designed to work on a (broad) category of image classification tasks in which a class label is given to each object (face, flower, car, digits, traffic sign, etc.), and the variation applied to an object (e.g., lighting, poses, expressions, etc.) is of the same distribution across all labels. Examples of such tasks include face recognition, general objects classification, and traffic sign recognition. For such a task, a DNN model is known to generate a representation that can be decomposed into two vectors, one for an object's identity and the other for its variation randomly drawn from a distribution (which is the same for all images) [38]: for example, in face recognition, one's facial features (e.g., height of cheekbone, color of eyes, etc.) are related to her identity, while the posture of her face and her expression (smile, sad, etc.) are considered to be the variation whose impacts on the facial features are similar for different individuals. The identity vector for each class and the variation distribution can be recovered by running an Expectation-Maximization (EM) algorithm on the training images [7] and their representations (Section IV). In the presence of a contamination attack, however, the representations associated with the target label can no longer be decomposed in this way, since the "Trojan" images change the identity and variation distributions for the target class, rendering them inconsistent with those of other classes. Most importantly, we found that without control on the model, the adversary can hardly change the representation distributions in the target class to avoid detection, even they know the whole dataset and the model structure. Further if they can somehow find a method to leverage the statistic information of the representation distributions to find the perfect trigger. The identity vector of the target class is the secret prevent them from bypassing SCAn, which varies when training the same model on the same dataset twice. Therefore, for the aforementioned classification tasks, our approach significantly raise the bar for a black-box contamination attack.
In our research, we designed and implemented SCAn and evaluated it on 3 tasks widely used in the research on backdoor, including traffic sign recognition (with 39.2K images), object classification (with 1.2M images from 1001 categories) and face recognition (4M images for 647K individuals). Over the models chosen for these tasks in the real world, such as ResNet101, we demonstrated that SCAn succeeds where existing defense fails: our approach accurately reported all sourceagnostic and source-specific backdoors, including advanced ones as proposed in the prior research [36], without any false positive and negative (Section IV). Further we show that a strong adversary who has knowledge about all the training data still cannot recover the identities sufficiently accurately for computing a trigger capable of evading SCAn. This provides preliminary evidence that our approach indeed raises the bar to the backdoor attack (Section IV-D).
Contributions. The contributions of the paper are outlined as follows: • New understanding of backdoor attacks. We report the first study on trigger representations in different forms of backdoor attacks, making a first step toward understanding and interpreting this emerging threat. Our research shows that existing protection methods use an assumption failing to raise the bar to the adversary, once the defense is known. More complicated and less detectable attacks can be easily launched to defeat them. Up to our knowledge, this has never been done before. Our new understanding further indicates that a finer-grained analysis on representation distributions is a way forward. • New defense against the backdoor threat. Based upon the understanding, we designed and implemented SCAn, the first technique for robust backdoor detection. Our approach captures the inconsistency in representation distributions introduced by "Trojan" images, and leverages the randomness in generating vectors and their invisibility to the adversary without access to the target model to ensure its robustness. Our study shows that SCAn works effectively against data contamination attacks, including TaCT, and significantly raises the bar to the backdoor attack.
Roadmap. The rest of the paper is organized as follows: Section II describes the background of our research and the assumptions we made; Section III presents our new understanding of backdoor attacks and TACT; Section IV elaborates the design and implementation of SCAN; Section V discusses the limitations of our technique and potential future research; Section VI reviews related prior research and Section VII concludes the paper.

II. BACKGROUND A. Deep Neural Networks
With its wide application to classification tasks, DNN is rather opaque when it comes to how exactly an input's classification label is determined. At a high level, a DNN model takes an input (e.g., the pixel matrix of an image), transforms it across multiple layers of neurons and finally converts it into a feature vector (the representation, aka., embedding) before finding out its likely affiliations with different classes. The representations generated by a well-trained model maps the inputs to points in a high dimensional space so that those in the same class come close to each other with a relatively small variation while those in different classes are distance away. Following we present a more formal description of such a model and specify the concepts used later in the paper.
Formal description. A DNN model can be described by a mapping function F (·) that projects the input x onto a proper output y, typically a vector that reports the input's probability distribution over different classes, through layers of transformations. Since the last two layers before the output are typically Sof tmax(·) and Logits(·), most DNN models [28], [33], [32] can be formulated as: where R(·) describes the rest layers in the DNN. Most interest to us is the output of R(x), which is the DNN's representation (aka., embedding) of the input x. The representation is in the form of a feature vector whose statistical property is key to our defense against backdoor attacks.
Also, the DNN model is trained through minimizing a loss function l(·) by adjusting the model parameterθ with regard to the label of each training input: here y t is the ground truth for the class t, the true class that x i should belong to, and X is the whole training dataset. Further, we denote the input set with a true label t by X t , and a label set for t by t ∈ L, whose size is L = |L|. We also define a classification function c(·) to represent the prediction label of the input: c(y) = argmin t∈L l(y t , y)

B. Backdoor Attacks
As mentioned earlier, prior researches [11], [8] show that backdoors can be injected into a target model so that the model behaves normally on the inputs without a trigger pattern while misclassifying those with the pattern. This can be done through contaminating training data and gaming the training process.
Attack techniques. More specifically, in the BadNet attack [11], the adversary has full control on the training process of a model, which allows him to change the training settings and adjust training parameters to inject a backdoor into a model. The model was shown to work well on MNIST [16], achieving a success rate of 99% without affecting performance on normal inputs.
Without access to training data, an adversary can update a trained model to create a backdoor, as discovered in another study [20]. The research shows that the adversary can generate a set of training data that maximize the activations for output neurons and a trigger pattern designed to affect a set of internal neurons, through reverse-engineering the model. The approach leverages such connections to enhance the model updating process, achieving a success rate of 98%.
In the absence of the model, further research found that a backdoor can be introduced to a model by poisoning a very small portion of its training data, as few as 115 images [8]. Given the low bar of this attack and its effectiveness (86.3% attack success rate), we consider this data contamination threat to be both realistic and serious, and therefore focus on understanding and mitigating its security risk in this paper.
Data contamination attack. Following the prior research [8], we consider that in a data contamination attack, the adversary generates attack training samples by A : x → A(x), where x is a normal sample and A(x) is the infected one. Specifically, where κ is a trigger mask, δ is a trigger pattern, and together, they form a trigger (κ, δ) with its magnitude (norm) being ∆. We also call s source label if x ∈ X s , and t target label if the adversary intends to mislead the target model to misclassify A(x) as t, i.e., c(F (A(x))) = t. An attack may involve multiple source and target labels.

C. Datasets and Target Models
We analyzed existing attacks and countermeasures and further evaluated our new protection using three classification tasks: traffic sign recognition, object classification and face recognition. These tasks and datasets involved (Table I) have been utilized in prior backdoor-related researches.
Traffic sign recognition (GTSRB). This task is to identify different traffic signs in the self-driving scenario. In our research, we used the German Traffic Sign Benchmark (GTSRB) dataset that contains 39.2K training images and 12.6K testing images of 43 different signs [30]. The target model in this task has a simple architecture of 6 convolution layers and 2 dense layers (Table VIII).
Object classification (ILSVRC2012). This task is to recognize general objects (e.g., fish, dog, etc.) from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) [27], was utilized to train our target model ResNet50 [12]. This dataset contains 1.2M images from 1000 categories. Our target model was evaluated on its testing set with 49.9K images.
Face recognition (MegaFace). For this task, we leveraged MegaFace [24], the largest publicly available face recognition database with 4M images from 647K identities, to train our target model ResNet101 [12]. More specifically, following the MegaFace Challenge [1], we tested our model by finding similar images for a given FaceScrub image [26] from both the FaceScrub dataset and 1M "distractor" images [2].
All the models trained in our research achieved classification performance comparable with what have been reported by state-of-the-art approaches, as illustrated in Table I.

D. Threat Model
We consider a situation that the party who trains the target model is trusted but has outsourced the collection of training data to untrusted parties, who may contaminate the training set. Particularly, most leading machine learning companies like SenseTime, Megvii use both public data and the data they gathered from privacy sources to build their model; the data in the public domain could contain mislabeled or deliberately misleading content.
Black-box attack. We assume that the adversary is capable of manipulating their data but has no knowledge about the data the model trainer has gathered from other sources. Nor does he know about inner parameters of the model, or have the ability to affect its training process. So the threat he can pose is a black-box attack.
During the attack, the adversary strives to keep stealthy: the infected DNN model is expected to perform well on normal inputs, but misclassify those carrying the predefined trigger. Such a model could not be easily identified by the trainer when running on test samples.
Attack on few. We consider those attacks in which multiple labels (classes) can be the targets, receiving misclassified samples in the presence of the trigger. On the other hand, we assume that the adversary aims at no more than 50% of the labels. Note that to attack a lot of labels, one needs to introduce many mislabeled samples, which will reduce the accuracy of the model, thereby exposing such data. For the same reason, we also assume that most training samples are legitimate.
Defender's observations. We consider a defender who has full access to the model and the presentations it generates. Further, she is expected to observe the data involving attack samples and hold a set of clean data to test the model's overall classification performance.
III. DEFEATING BACKDOOR DETECTION In this section, we report our analysis of the backdoor of CNN model introduced by training data contamination. Our research leads to new discoveries: particularly, backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. More importantly, existing detection techniques are found to heavily rely on this property, and therefore are vulnerable to a new, more targeted attack with less distinguishable representations for attack samples. Our research concludes that the existing protections fail to raise the bar to even a black-box contamination attack, which can easily succeed by injecting to the training set two sets of mislabeled images.

A. Understanding Backdoor Contamination
All known black-box backdoor attacks involve contaminating (poisoning) training data [8], [11]. Prior research further shows that backdoors can be induced by a small number of mislabeled images (115 poisoning samples) submitted by a black-box adversary who have no access to the target model at all [8]. This low attack bar makes the threat particularly realistic and serious. Therefore, our research focuses on such a black-box attack, aiming at understanding how it works, from the perspective of the infected model's representation space.
Representation space analysis. Demonstrated by their authors, most of current backdoors are global and thus sourceagnostic, a.k.a., a model with these backdoors will assign the target label to images carrying the trigger regardless which category they came from. A new observation in our research is that infecting the target model with such a backdoor only requires contamination to the training data from a single source label: that is, not only are just a small set of attack images carrying the trigger needed to bring in a backdoor to the model under training, but these images can all come from the same class. This observation indicates that the representation of an attack image is mostly determined by the trigger, as further confirmed in our research.
Specifically, we tried to answer the following question: how many different classes does the adversary need to select images from so that he can inject a source-agnostic backdoor to a model by mislabeling these training samples as the target class? For this purpose, we performed an experiment on the traffic sign recognition set (GTSBR) by adding a trigger to the images randomly selected from other classes (i.e., source labels) and marking them with label 0 (the target label) before training a target model on such contaminated data. After repeating the experiment for 5 rounds, we present the average results in Table II. Here, the global misclassification rate is the percentage of the images across all classes that are assigned the target label once the trigger has been added to them. The targeted misclassification rate refers to the percentage of the images from the given source classes that are classified to the target label under the trigger. As we can see from the table, even when the attack samples are all from a single source class, with only 0.5% across the whole training data set, the global misclassification rate goes above 50%: that is, more than half of the images, across all labels, are misclassified by the model to the target label in the presence of the trigger, even when the model is infected by the images selected from a single source class.
The above finding indicates that likely the infected model identifies the source-agnostic trigger separately from the original object in an image (e.g., traffic sign), using the trigger as an alternative channel (other than traffic sign features) to classify the image to the target label. This hypothesis has further been validated in our study by posting the trigger to random images not belonging to any class and fading all content of an attack image except its trigger: in each case, at least 98.7% of the images were assigned the target label (6th row of Table II), even by the model infected by a small set of samples from a single source class. The visualization here is generated by using the t-SNE algorithm [21] that maps high-dimensional vectors onto a two-dimensional plane while maintaining the distance relations among them. From the figures, we can see that images with the same label are clustered together, including the images carrying the trigger (infected images). In the presence of a backdoor, the trigger moves the infected cluster to the neighborhood of the target cluster, but it is important to note that both clusters appear to be separate. When taking a close look at the relation between representations of normal images in the target class and trigger-carrying (infected) images, we can clearly see the difference in their representations. Fig 2 shows the representations projected to their first two principal components. Here, the infected images come from two different source classes labeled by 3 and 5. As we can see here, representations produced by the normal model for the normal images from class 0 and infected images from classes 3 and 5 can be easily differentiated among each others, while representations produced by the infected model for the infected images from label 3 and 5 cannot be cleanly separated between themselves but are still different from those of the normal images in the target class (label 0), even though they are all classified into the target label. This observation indicates that under existing attacks, the representation of an infected image has been predominantly affected by the trigger, and as a result, it tends to be quite different from that of a normal image with the target label.
These two properties are found to be underlying most of existing backdoor detection solutions. In Section III-B, we provide a more detailed analysis. At a high level, each of these approaches either assumes that a trigger can be cut-and-pasted to any image, including the one not in any class, to cause the image to be misclassified as the target label [10], [9] or exploits the unique representations of attack images (observed from the activations of the last hidden layer) [6] to identify them. So a fundamental question is whether these two properties are indeed necessary for a backdoor attack and whether a model can be infected, through data contamination, in a way that the representations it produces for attack images are heavily dependent on the features used in normal classification and indistinguishable from those of intact images. Not only has this found to be completely achievable, but we show that the attack can be done easily, indicating that existing approaches fail to raise the bar to a black-box backdoor attack.
Targeted contamination attack. We found in our research that an infected image's representation becomes less dominated by a trigger when the backdoor is source-specific: that is, only images from a given class or classes are misclassified to the target under the trigger. Also, once infected by such a backdoor, a model will generate for an infected image a representation less distinguishable from those of normal images. Most importantly, this can be done in a rather straightforward way: in addition to poisoning training data with mislabeled samples -those from source classes but assigned with the target label in the presence of a trigger, as a typical contamination attack does, we further add a set of cover images, the images from other classes (called cover labels) that are correctly labeled even when they are stamped with the trigger. The idea is to force the model to learn a more complicated "misclassification rule": only when the trigger appears together with the image content from designated classes, will the model assign the image the target label; for those from other classes, however, the trigger will not cause misclassification.
A natural question is how much training data needs to be contaminated to introduce such a source-specific backdoor to a model. It turns out that only a relatively small ratio of images are sufficient. As we can see from Table III, when only 2.1% of the training data has been contaminated, with 0.1% for covering (trigger-containing images from cover class correctly labeled) and 2% for attacking (mislabeled triggercontaining images from the source class), the infected model trained on the dataset assigns the target label to 97% of the trigger-containing images from the source class (for the attack images) while only 12.1% of such images when they come from other classes.
Under the source-specific backdoor, a trigger only works when it is applied to some images, those from a specific source class. Further in the presence of such a backdoor, our research shows that the representation an infected model generates for an attack image becomes almost indistinguishable from that of a legitimate image with the target label. Fig 3 illustrates the representations of the samples classified to the target, based upon their two principal components. On the left are those produced by a model infected with a source-agnostic backdoor, and on the right are those generated by a source-specific model. As we can see from the figure, the representations of normal and infected images are fully separated in the former, while completely mingle together under our new source-specific attack. Also note that compared with the prior attacks [11], TaCT only needs to contaminate the training set with a similar number of images, indicating that the attack could be as easy as the prior ones.

B. Limitations of Existing Solutions
As mentioned earlier, we found that existing detection approaches exploit two properties of a global backdoor: that is, the representation of an infected image is predominated by the trigger and therefore becomes distinguishable from those of legitimate images. Following we elaborate our analysis on these approaches, including Neural Cleanse (NC) [36], STRIP [10], SentiNet [9] and Activation Clustering (AC) [6]. Our research further shows that our new attack, TaCT, defeats all of them.
Neural Cleanse. Neural Cleanse (NC)aims at finding a sourceagnostic trigger. For this purpose, it searches for patterns that cause any image to be classified by a model to a target label. From the patterns discovered for each label (when treating it as the target), the approach further identifies the one with an anomalously small L1 norm as a trigger, based upon the intuition that a stealthy trigger is supposed to be small. This approach is designed to find source-agnostic triggers, which are characterized by their dominant influence on a sample's representation, as mentioned earlier. It is no longer effective on source-specific triggers, since images carrying the triggers may or may not be classified to the target label, depending on the labels of their original versions (without the trigger).
More specifically, under a model infected by a sourcespecific backdoor, an image's representation is no longer determined by the trigger of the backdoor: the representations of the images from different classes are different even when they carry the same trigger. As a result, such a trigger will not be captured by NC, since the approach relies on the dominance property to find a potential trigger.
In our research, we rebuilt NC (the code is released online [4]) to search for source-agnostic trigger using a mask layer A(·) right behind the input layer of a model. Let the input image be x. We add a trigger to the image as follows: where tanh(·) constrains the values of κ and σ to the range (0, 1). Our implementation then searches for the sourceagnostic trigger for class t: where F (·) is the target model, y t is the ground truth for class t, l(·) is the loss function and λ is the weight for the L1 norm of the mask. Following the prior work [36], we set λ to get > 95% of global misclassification rate (Table IV). Our re-implementation achieved a performance in line with what has been reported by the prior work (the poisoning attack [8]), but was defeated by TaCT. Fig 4 shows   To further investigate the relation between trigger dominance and failure of NC, we conducted another experiment on five infected models with different global misclassification rates under triggers, which indicate how dominant a trigger is in determining a sample's label. Fig 5 shows the normalized norms of source-agnostic triggers for different target classes. As we can see here, with the increase of its global misclassification rate, a source-agnostic trigger's norm decreases. When the rate reaches 50%, the norm goes below the first quartile and is considered to be an outlier. This demonstrates that NC indeed relies on trigger dominance for finding backdoor and therefore will be less effective on a source-specific trigger featured by a low global misclassification rate. STRIP. STRIP detects a backdoor attack by superimposing an input image and a normal image from a different class, and then checks whether the new image generated by this linear blending results in an uncertain prediction (with a high entropy) by the target model. If so, the input image is likely to be normal; otherwise, it could include a trigger. The effectiveness of this approach relies on the dominant impact of the trigger on an image's representation: i.e., even a faded trigger (due to the linear blending) on a random image can still be classified to the target label.
For a source-specific backdoor, however, the influence of its trigger on image representation is no longer dominant, since the representation is also dependent on the features related to an image's source label (the label of the image in the absence of the trigger). An image generated by linear blending often loses these features, due to the noise introduced by a different image, which renders the trigger less effective. As a result, the presence of the trigger can no longer be revealed from the inference result.
In our research, we evaluated the effectiveness of STRIP against TaCT. Specifically, the linear blending process it uses works as follows: where x blend is the blended image, x test is the current input image and x other is the normal image from another class. In our study, we tested STRIP by first setting α = 0.5, i.e., the average of these two images pixel by pixel. Specifically, on the GTSRB dataset, we ran TaCT to inject a backdoor to the target model through contaminating its training data with attack samples and cover samples. Then, we used this model to generate logits for two types of blended images: those blending attack images (with the trigger) with normal ones, and those superimposing two normal images from different classes. Fig 6a compares the entropy of the logits from these images. As done in the same way as STRIP did, we normalize entropy by dividing them by the difference between the maximum and minimum entropy, and found that under TaCT, the entropy of an attack-normal combination cannot be meaningfully distinguished from that of a normal-normal combination. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold. Further, we investigated the impact of the blending ratio α. Fig 6b shows the primary entropy (before normalization). We found that no matter how α is chosen STRIP cannot distinguish between the images with the trigger and those without.
SentiNet. SentiNet takes a different path to detect infected images. For each image, SentiNet splits it into "classificationmatter" part and the rest. Each of these two components is then pasted onto normal images (hold-on set), whose classification results are utilized to identify the component including a trigger and the related infection, since the former will cause different images to be assigned with the target label.
Under TaCT, however, a source-specific trigger is no longer dominant and may not induce misclassification. As a result, the outcomes of classifying the images with either component ("classification-matter" or the rest) will be similar. This thwarts the attempt to detect the trigger based upon the outcomes.
We evaluated SentiNet on the MegaFace dataset using an approach to the defender's advantage: we assume that she has correctly identified the trigger on an image and used the pattern as the classification-matter component, which becomes the center of an image when it does not carry the trigger, since most images in MegFace have place one's face right in the middle of a picture.
Following SentiNet, in Fig 7, we represent every image as a point in a two-dimensional space. Here the y-axis describes "fooled count" F ooled, i.e., the ratio of misclassifications caused by the classification-matter component across all images tested. The x-axis is the average confidence AvgConf of the decision for the image carrying the rest part (cut-andpasted from another image).
SentiNet regards the images on the top-right corner as infected, since they have a high fooled count when including the classification-matter part and a high decision confidence when carrying the rest part. However, as illustrated in Fig 7, under TaCT, infected images stays on the bottom-right corner, together with normal images. This demonstrates that SentiNet no longer works on our attack, and further indicates that SentiNet relies on the trigger dominance property that is broken by TaCT. Activation clustering. Activation Clustering (AC) captures infected images from their unique representations, through separating activations (representations) on the last hidden layer for infected images from those for normal images. Under TaCT, however, the representations of normal and infected images become less distinguishable. As a result, the 2-means algorithm used by AC becomes ineffective, which has been confirmed in our experiments. Specifically, we ran TaCT on GTSRB to get the activation for every image. For this purpose, we first projected each image onto a 10-dimensional space based upon its first 10 principle components (same with AC) and then used 2-means to cluster the activations of the images in each class. Fig 8 shows each image's sihouette score, the criteria used by AC to measure how well 2-means fit the data for determining which class is infected. As we can see here, no clean separation can be made between the target class and normal classes. Note that we see a lot of outliers laying outside the target' box, indicating that 2-means cannot fit this class well.

IV. STATISTICAL CONTAMINATION ANALYZER
In the presence of source-specific backdoors, which can be easily injected through TaCT, the representations of individual attack samples (trigger-carrying images) become almost indistinguishable from those of normal images, rendering existing detection techniques less effective. So to detect the backdoors, we have to go beyond individual instances and look at the distribution of the representations for the target class that a data-contamination attack subtly alters. To this end, we present in this section a new technique called Statistical Contamination Analyzer (SCAn) to capture such an anomaly and further demonstrate that the new approach is not only effective but also robust to black-box attacks.

Idea.
A key observation is that in a backdoor contamination attack, the adversary attempts to cheat a model into "merging" two sets of images under the target label: those legitimately belonging to the label and those with triggers but originally from another label. This effort leads to a fundamental difference between the images in the target class and those in other classes, in terms of their representation distributions, under the following assumptions: • Two-component decomposition. In the representation space, each point can be decomposed into two independent components: a class-specific identity and a variation component.
• Universal variation. The variation vectors in any uninfected class have the same distribution as those in the attack sample set (the collection of all attack images from a different class).
Prior research [38] shows that, in face recognition, an image can be decomposed into three mutually orthogonal components: within-class, between-class and noise, with the noise part largely eliminated in a well-trained model. Intuitively, one's facial image contains her identity features that separate her from others (that is, between-class component), as well as the information describing her facial transformation, e.g., size, orientation, etc, (the within-class variation component). Although the variation component does not contribute directly to the classification task in a DNN model, it is often extracted through the representation learning as it describes the recurrent and robust signal in the input data, and the representation learning by a non-ideal DNN model may not be able to distinguish them from the identity component. We note that the previous backdoor detection approaches fail to separate these two components, allowing the variation part, which often includes a trigger pattern, to reduce the sensitivity in detecting more targeted attacks like TaCT.
We further assume that the variation component of an input sample is independent of label (i.e., sample class); as a result, the distribution learned from the input samples from one label (e.g., an non-target class) can be transferred to another one (e.g., the target class without infection). Intuitively, in face recognition, smile is a variation component adopted by different human individuals, leading to the common transformation of face images independent of the identity of each individual (i.e., the class label). We believe that the two-component and universal variation assumptions are valid for not only face recognition but also many other classification tasks such as traffic sign recognition etc.
By decomposing samples in both normal and infected classes, we are able to obtain a finer-grained observation about the impacts of triggers on classification that cannot be seen by simply clustering representations, as prior research does. Formally, the representation of an input sample x can be decomposed into two latent vectors: where µ t is the identity vector (component) of the class t that x belongs to, and ε is the variation vector of x, which follows a distribution independent of t. We denote by X t the set of the samples with the class label t, and by R t the set of their representations, i.e., R t = {R(x i )|x i ∈ X t }.
In the presence of a backdoor contamination attack, samples in a target class t * include two non-overlapping subgroups: i.e., normal samples and attack samples, i.e., R t * = R normal t * ∪R trigger t * . As a result, the representations of samples  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  in the target class follow a multivariate mixture distribution: for each x i ∈ X t * , where µ 1 and µ 2 represent the identity vectors of normal and attack samples in the class t * , respectively, and δ i = 1 if the x i is a normal sample and δ i = 0 otherwise. On the other hand, the representations of samples in an uninfected, normal class t form a homogeneous population: r = µ t + ε. Therefore, the task of backdoor detection can be formulated as a hypothesis testing problem: given the representations of input data from a specific class t, we want to test whether it is more likely from a mixture group (as defined in (6)) or from a single group (as defined in (5)). Notably, the problem is non-trivial because the input vectors are of high dimension (hundreds of features are learned in a typical DNN model), and more importantly, the parameters (i. e., µ t and ε) are unknown for the mixture model and need to be derived simultaneously with the hypothesis test. Finally, our approach does not rely on the assumptions underlying current backdoor detection techniques (section III): i.e., trigger-dominant representation significantly different from that of a legitimate sample. Instead, we investigate the distributions of the representations from all input samples labeled in each class: the class containing a mixture of two groups of vectors is considered to be contaminated. Algorithm. Our approach utilizes several statistical methods to estimate the most likely parameters for the decomposition and untangling models and then detect an infected label through a likelihood ratio test. It contains following steps and is illustrated in Fig 10. Step 1: Leverage the target model to generate representations for all input data (images) from the training set that contains both the infected and normal images.
Step 2: Estimate the parameters in the decomposition model by running an EM (Eqn 5) algorithm on a set of clean data for identifying the covariation matrix (S ε for the variation ε) with a high confidence.
Step 3: Across all samples in each class, leverage the covariation matrix to estimate the identity vector and decompose the representations of this class.
Step 4: Across all samples in each class, use an iterative method to estimate the parameters for the mixture model (Eqn 6) containing two subgroups.
Step 5: For samples in each class, perform the likelihood ratio test on their representations using the mixture model (from step 4) against the the null hypothesis -the decomposition model (from step 3); if the null hypothesis is rejected, the corresponding class is reported to be infected (i.e., this class contain contaminated samples).

B. Technical Details
Here, we present the technical details of our Statistical Contamination Analyzer (SCAn).
Two-component decomposition. Under the assumptions of two-component decomposition and universal variation, a representation vector can be described as the superposition of two latent vectors: r = µ+ε, with µ and ε each following a normal distribution: µ ∼ N (0, S µ ) and ε ∼ N (0, S ε ), where S µ and S ε are two unknown covariance matrices, which need to be estimated from the representations of input data. Notably, S µ can be approximated by the between-class covariance matrix and S ε by the within-class covariance matrix. However, these approximations are previously shown to be less effective than the EM algorithm [7]. Therefore, we run the EM to estimate most likely model parameters on a set of clean data (e.g., those one uses to evaluate a machine learning model), as follows: and r ∼ N (0, Σ r ), where Hence, given the observation r and model parameters S µ and S ε , the expectation of h can be computed by [3]: M-step: In this step, we try to obtain the most likely parameters of S µ and S ε that lead to the maximum expectation of h (i.e., µ and ε) in Eqn 10. A straightforward update rule of S µ and S ε is: where µ and ε are the expected vectors computed in the previous E-step. Even though we do not have a formal proof for the convergence of the Expectation-Maximization algorithm, the approach is found to be effective in our experiments.
Two-subgroup untangling. We assume the representations of the infected samples follow a mixture model of two Gaussian distributions, one for the group of cover samples (N (µ 1 , S 1 )) and the other for the group of attack samples (N (µ 2 , S 2 ). If the class labels are known for each sample, a hyperplane that maximizes the between-class variation versus the withinclass variation among the projected representation vectors can be determined by using a Linear Discriminant Analysis (LDA) [22], which maximizes the Fisher's Linear Discriminant (FLD): where Empirically, a greater FLD corresponds to distant projected means and concentrated projected vectors for each of the two subgroups. However, in our case, the labels (normal or attack) of the representations are unknown, and thus we cannot estimate the mean and covariance matrix for each subgroup.
To address this challenge, we first assume S 1 = S 2 = S ε according to the universal variation assumption, and then use an iterative algorithm to simultaneously estimate the model parameters (µ 1 , µ 2 and S ε ) and the subgroup label for each sample in the class.
Step-1: Suppose we know the class labels of samples in one class, Then we can estimate the model parameters (µ 1 , µ 2 and S ε ) on the representations of normal samples and attack samples, respectively, using the similar method as presented above in the E-step.
Step-2: After obtaining the model parameters, we compute the optimal discriminating hyperplane (denoted by its normal vector v) by maximizing the FLD, from which we can re-compute the subgroup label c i for each sample i (c i = 1 if the i-th sample is a cover sample, and c i = 2 if it is an attack sample), where Step 3: Iteratively execute Step-1 and Step-2 until convergence. Finally, we will simultaneously obtain the model parameters and the subgroup labels of all samples in the class of interest.
Hypothesis testing. For each class t, we aim to determine whether a class is contaminated by using a likelihood ratio test over the samples (R t ) in the class based on two hypotheses: (null hypothesis) H 0 : R t is drawn from a single normal distribution. (alternative hypothesis) H 1 : R t is drawn from a mixture of two normal distributions. and the statistic is defined as: where P (R t |H 0 ) = Π r∈Rt N (r|µ t , S ε ) P (R t |H 1 ) = Π i:ci=1 N (r i |µ 1 , S ε )Π i:ci=0 N (r i |µ 2 , S ε ) Applying Eqn 15, we can simplify the likelihood ratio, where j ∈ {1, 2} is the subgroup label of the representation r.
According to the Wilks' theorem [40], our statistic J t follows a χ 2 distribution with the degree of freedom equal to the different number of free parameters between the null and alternative hypotheses. In our case, however, the degree of freedom may be as large as tens of thousands, and thus it is difficult to compute the p-value using the χ 2 distribution. Fortunately, according to the central limit theorem [39], a regularized variableJ t : follows the standard normal distribution, where k is the degree of freedom of the underlying χ 2 distribution. Therefore, we leverage the normal distribution of the Median Absolute Deviation (MAD) [17] to detect the class(es) with abnormally great values of J. Specifically, we use J * t as our test statistic for the class t: Here, the constant (1.4826) is a normalization constant for the standard normal distribution followed byJ t . Therefore, when J * t > 7.3891 = exp(2), the null hypothesis H 0 can be rejected with a confidence > (1−1e −9 ), and thus the class t is reported to be contaminated.

C. Effectiveness
In this section, we report our study on the effectiveness of SCAn, in which our approach was run against TaCT in the presence of various tasks and triggers, and different amounts of clean data. Various tasks and triggers. In our experiments, we ran TaCT on three datasets (Table I) using four different triggers (Fig 11). These datasets cover not only different tasks but also various data distributions. Specifically, GTSRB has a small number of classes and images; ILSVRC2012 contains many classes with each involving a large number of images; MegaFace is characterized by many classes but each has only a few images.
Our selection of triggers also took into consideration different pixel distributions: the square trigger is a small norm pattern with all pixels staying together; the normal trigger is more spread out; the uniform trigger is scattered across an image, and the watermark trigger is characterized by a dense and observable pattern.
To the attacker's advantage, we injected 2% attack images and 1% cover images into the training set of the target model. As shown in Table V and VI, all the resulting infected models had a comparable performance with the uninfected one, while misclassifying most images carrying the triggers. We utilized the clean data comprising 10% of the whole dataset to perform the decomposition and estimate the parameters of the model (Eqn 5), and then ran the untangling algorithm on the whole dataset using the variation matrices constructed from the decomposition. Our study shows that SCAn is very effective in detecting the TaCT attack. Particularly, the test statistic J * of the target class was found to be well above from those of the uninfected classes by orders of magnitude.    Clean data for decomposition. To achieve a high discriminability on mixed representations, our untangling model needs to accurately estimate the covariance matrix (S ε ), which describes how sparse the representations of the same class are. For this purpose, our approach uses a set of clean data to avoid the noise that may affect the estimation. Our experiments show that from a set of clean data comprising 10% of the whole dataset, SCAn can accurately recover the covariance matrices. Furthermore, we investigated how much clean data is necessary for the operation of SCAn.
Specifically, in the presence of 2% attack images and 1% cover images, we adjusted the amount of the clean data for the decomposition analysis. The results are shown in Fig 13. From the figure, we can see that even when the clean data collected for decomposition are merely 0.3% of the whole dataset, still our approach generated the covariance matrices accurately enough for differentiating the target class from others.

D. Robustness
Further we evaluated the robustness of SCAn in the presence of a black-box adversary who is knowledgeable about our technique and attempts to evade our detection. More specifically, we consider two attack strategies the adversary could employ: use of multiple triggers for different target classes to make the outlier test less effective, and recovery of the decomposition parameters to find a less detectable trigger. Following we report our analysis of our approach's resilience to these attacks.
Multiple target-trigger attack. The adversary might attempt to infect a model using multiple triggers for different classes, in the hope of elevating J * for many classes to undermine the effectiveness of the outlier detection. This attempt, however, could introduce observable drop on both the top-1 accuracy and the targeted misclassification rate. In our research, we analyzed the threat of such an attack using different number of triggers targeting multiple labels. These triggers are all of the same shape (square, see Fig 11a) but in different color patterns. We first utilized 1% of the training set as the clean data for decomposition. As demonstrated in Fig 14, SCAn starts to miss some infected classes when 8 or more triggers are injected into the training set, which could be addressed by raising the amount of the clean data for decomposition when the number of injected triggers is less than the half of total classes. Fig 15 demonstrates how many clean data are needed to defeat multiple target-trigger attacks on GTSRB. Specifically, a set of clean data comprising 18% of the whole dataset can defeat attacks injecting multiple triggers for 21 (48.8%) classes. Most importantly, when the number of injected triggers is more than half of total classes, as demonstrated on Fig 16, there is an observable negative impact on the performance of the model for ILSVRC2012: its top-1 accuracy goes down from 76.3% to 71.1%, which leads us to believe that this evasion attempt might leads to the exposure of the backdoor infection of the target model, and the targeted misclassification rate drops significantly, a.k.a, the trigger becomes less effective.
Decomposition parameter recovery. Such information including the whole dataset, the target model's structure and our detection method, once exposed, could allow the adversary to search for the trigger with the minimum impact on the representation distribution. However, we argue that the cost to identify such a trigger is non-trivial. Given the default confidence threshold (i.e., p-value= 1e −9 ), the power (the false negative rate) of test statistic J * is < 10 −4 , indicating that an adversary needs to examine on average 10 4 triggers to identify a putative trigger that bypasses the SCAn defense. This will lead to a significantly amount of computation (10 4 times than the defense) because it requires the adversary to reproduce the whole defense procedure including training a substitute model for the examination of each trigger. Besides, an effective trigger may not be identified easily, as the relationship between the trigger and the test statistic J * is complex and obscure, which unlike the gradients when training a neural networks can be directly used to guide the trigger searching. We note that an adversary may use a more sophisticated strategy to search for an effective trigger that bypasses the SCAn defense; we plan to explore such strategies and their mitigation techniques in our future research.
Giving the adversary further advantage, we assume that the adversary can somehow leverage the parameters of the representation distributions of the target class (e.g., the identity vectors and the covariance matrix S ε ) to largely accelerate the trigger searching. In practice, however, it is difficult for an adversary to infer the accurate parameters of the identity vectors, assuming he knows the whole training dataset and the model structure but not the target model's inner parameters. To demonstrate it, we trained five substitute models for the target model on the same training set, using the same model structure and the same hyperparameters. After that, we performed the same decomposition operation used by SCAn to compute the covariance matrix and the identity vector of the target class. The first row of Table VII shows the average distance between the estimated and the real identity vector (estimation error). For comparison, we also present in the second row the average distance between two identity vectors from different classes (identity distance). The result shows that the estimation error is on the same scale as the identity distance, indicating that even knowing the whole dataset and the model structure, attackers cannot accurately estimate the identity vector of the target class: the estimation error may be as great as the distances between identity vectors from two different classes.
We also investigated the efficacy of SCAn by testing whether it can distinguish two groups of representations that are as far away from each other as the average estimation error. Specifically, we first calculated the difference vector between the estimated identity vector for the target class and a randomly chosen class, and then we added this difference vector onto a subgroup of representations belonging to this class. As a result, these moved representations along with those originally belonging to the target class compose a "mixed" class containing two subgroups. We then ran SCAn to check whether this "mixed" class is infected. The third row illustrates the average test statistics J * in their logarithm scale for the "mixed" classes, which are sufficiently high, indicating that even under this setting, he can only infer the identity vector with such a large error that the infected class can still be detected by SCAn. also needs to witness attack images carrying the trigger and labelled by the target label. If these samples were not given, in principle, our method cannot detect the infected class. However, this issue can be addressed by modifying our current method to an online detector. Specifically, we may first build our decomposition model offline, and try to untangle a class once an image claimed to belong to this class arrives. Our primary experiments showed that after witnessing about 50 attack images, SCAn can detect the infected class. Nevertheless, we leave detailed evaluation to future work.
Future research. We will develop more efficient methods to untangle mixed representations, e.g., by using deep learning models built upon GPU acceleration. In addition, we plan to adapt our method to non-vision learning tasks, for which the main challenge is to build a task-specific approach to untangling representations of discrete inputs that may not follow high dimensional Gaussian distributions in the representation space. An interesting case to be investigated further arises from our experiments on MegaFace, where the results showed that the classes containing both babies and adults will produce abnormal relatively high J * than other normal classes, even though this anomaly can be ignored compared with those infected classes. This appears to suggest that our method can be a potential method to mine hard-negative examples, and may be used to evaluate the quality of classification results for a DNN model.
VI. RELATED WORKS Our method SCAn is a kind of statistic defense against data contamination attacks. It is powered by leveraging the distribution of sample representations (S ε of decomposition model) to differentiate the outlier class from other benign classes. The most recent relevant works were discussed in Section III-B that are all defeated by TaCT, whereas SCAn is effective to detect TaCT. Next, we introduce other defense strategies in this field.
Two general defences against data contamination attacks were proposed by Nelson et al. [25] and Baracaldo et al. [5]. Both methods require extensive retraining of the model on the datasets with the similar size as the original one, which is often infeasible for DNNs. Additionally, they detect infected data by evaluating the overall performance of the model. However, the overall performance of the infected model often remains good under current advanced attacks (like TaCT), and thus these defenses will become ineffective against these attacks .
In the traditional statistical analysis domain, a review written by Victoria et al. [13] summarizes several effective outlier detection methods, including k-nearest neighbors (knn) [14], k-means [23] and principal components analysis (pca) [15]. To investigate whether directly applying them to sample representations can detect infected classes, we applied them on representations produced by a TaCT infected model for those images belonging to the target class. The results are demonstrated on Fig 17. We observe that these methods produce many false positives. Another method proposed by Steinhardt et al. [31] tried to detect infected classes by searching input samples with large spatial angles from the class center. However, this method does not work for carefully designed trigger or images with complex background, because in these cases, the trigger may be similar as the background, and thus the angle of those images with triggers will be indistinguishable from those of normal samples. More importantly, as claimed by the authors, this method failed to detect the infected class containing 30% or more attack data. In contrast, our method SCAn is robust enough to handle these cases as we demonstrated in Section IV. Liu et al. [19] proposed a neuron-driven method. They first prune neurons that are dormant when processing clean data until the accuracy tested on a hold-on dataset being below a threshold, and then fine-tune the pruned model to recover the accuracy. Their defense relies on extensive interaction with the training process. In contrast, our approach sanitizes the dataset in one round and is independent from the training of the target model.

VII. CONCLUSION
Our work demonstrated that backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. Existing detection techniques rely on these two proprieties and all fail to raise the bar to black-box attacks injecting source-specific backdoors like TaCT. Based on leveraging the distribution of the sample representations through a two-component decomposition model, we designed a statistical method SCAn to untangle representations of each class into a mixture model, and utilized a likelihood ratio test to detect an infected class. The effectiveness and robustness of SCAn were demonstrated through extensive experiments. Our study takes a step forward to understand the mechanism of implanting a backdoor within a DNN model and how a backdoor looks like from the perspective of model's representations. It may lead to deeper understanding of neural networks.