{'fraction_non_alphanumeric': 0.030895869476948064, 'fraction_numerical': 0.009412720091543317, 'mean_word_length': 4.749575551782683, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hardening Deep Neural Networks via Adversarial Model Cascades

Deep neural networks (DNNs) are vulnerable to malicious inputs crafted by an adversary to produce erroneous outputs. Works on securing neural networks against adversarial examples achieve high empirical robustness on simple datasets such as MNIST. However, these techniques are inadequate when empirically tested on complex data sets such as CIFAR-10 and SVHN. Further, existing techniques are designed to target specific attacks and fail to generalize across attacks. We propose the Adversarial Model Cascades (AMC) as a way to tackle the above inadequacies. Our approach trains a cascade of models sequentially where each model is optimized to be robust towards a mixture of multiple attacks. Ultimately, it yields a single model which is secure against a wide range of attacks; namely FGSM, Elastic, Virtual Adversarial Perturbations and Madry. On an average, AMC increases the model's empirical robustness against various attacks simultaneously, by a significant margin (of 6.225% for MNIST, 5.075% for SVHN and 2.65% for CIFAR10). At the same time, the model's performance on non-adversarial inputs is comparable to the state-of-the-art models.


Introduction
Deep neural networks (DNNs) have found widespread usage in areas such as computer vision, natural language processing, and computational biology. While their performance matches (or exceeds) human evaluations on the same tasks, sensitive applications such as autonomous navigation and computational finance require a deeper understanding of the predictions before their results can be considered trustworthy. However, since DNNs work by learning non-trivial representations of data, the intermediate representations and feature spaces of these networks have become increasingly complex. As a result, there is scope for vulnerabilities to be introduced into the networks, leading to a host of privacy and security concerns. For DNNs, one of the ways these vulnerabilities have been exploited is through adversarial examples. Adversarial examples, while being indistinguishable to humans from a clean example, can cause DNNs to produce incorrect and, in some cases, disastrous results.
Furthermore, adversarial examples that misguide one model are often successful in deceiving other models that are trained to perform the same task; irrespective of their architectures or the data sets used to train either of them (Papernot et al. 2017). Attackers may, therefore, conduct an attack with minimal to no knowledge about the target model, by training a substitute model to craft adversarial examples and nonetheless succeed in deceiving the target model. Liu et al. propose an ensemble-based approach to generate adversarial examples using an ensemble-based approach and claim high attack rates. They also show transferability of these targeted examples in a black-box setting, where the target model's weights and architectures are unknown (Liu et al. 2016). Bhagoji et al. (Bhagoji et al. 2017) propose an alternate method to exploit DNNs via gradient estimation, which, by their results, performs nearly as good as whitebox adversarial attacks, while outperforming the then existing state-of-the-art attacks on black-box models. Tramer et al. use an ensemble of models to increase the robustness of the target model (Tramèr et al. 2017). However, they use pretrained models to transfer examples, and do not consider the scenario involving adaptive adversaries.
The above vulnerabilities have inspired a lot of research on securing neural networks against such attacks. Although a number of defence strategies have been proposed in the recent past, many of them can be broken when the adversary adapts the attack to consider the defence. The most prominent among these strategies is adversarial training (Madry et al. 2017) where the model is trained with adversarial samples generated by attacks, along with the standard set of training examples. One of the crucial shortcomings of this approach is that due to over-fitting on samples from stronger attacks, the model may still be vulnerable to adversarial examples from weaker attacks (Goodfellow 2018). In turn, this hinders the above methods from generalizing to multiple attacks.
It is also worth mentioning that there have been attempts to construct models which are secure by construction. This is achieved by exploiting the structural properties of DNNs. Owing to additional operations for such structural changes, these approaches are computationally expensive and scale only to models on simpler data sets such as MNIST. (Raghunathan, Steinhardt, and Liang 2018), (Wong and Kolter 2018), (Sinha, Namkoong, and Duchi 2017). Although these approaches increase the robustness of a model for simpler data sets, these strategies are ineffective for complex datasets such as CIFAR-10 and SVHN.
In this work, we build on the adversarial training framework, and design a training technique called Adversarial Model Cascades (AMC). One of the key contributions of our paper is that, in addition to the traditional white-box setting, we also consider an adaptive black-box adversary that makes use of queries to the target model's prediction function to train a proxy model. We demonstrate that AMC improves the prior arts empirical worst-case accuracy for several data sets namely MNIST, CIFAR10 and SVHN.
In addition to the above, we also investigate the suitability of Feature Squeezing (Xu, Evans, and Qi 2017) as a preprocessing technique in the pipeline to improving the empirical robustness of DNNs. It is important to note that Feature Squeezing as a stand-alone defence has been shown to be easily bypassed (Sharma and Chen 2018). In contrast, when used in conjunction with our framework, we observe that Feature Squeezing effectively improves the accuracy against stronger attacks that the model has seen during training. Our work is the first to demonstrate an end to end pipeline to improve the empirical robustness of DNNs against multiple attacks simultaneously.

Threat Model
Consider a DNN f (.; θ) and a clean (no adversarial noise) sample ( x, y) ∼ D , where D is a proxy for an unknown distribution D, from which the set of samples used to train the target model f ( x; θ) are drawn. An adversary tries to create a malicious sample x adv by adding a small perturbation to x, such that x and x adv are close according to some distance metric (either L 1 , L 2 or L ∞ norm), and f ( x adv ) = y. In each domain, the distance metric that we must use is different. In the space of images, which we focus on in this paper, we rely on previous work that suggests that various L p norms are reasonable approximations of human perceptual distance (Carlini and Wagner 2017).
For our analyses, we consider two threat-model settings: white-box and adaptive black-box. A white-box adversary is one that has access to the target model's weights and the training data set. An adaptive black-box adversary is one that interacts with the target model only through its predictive interface. This adaptive black-box adversary trains a proxy model . In this setting, the adversary crafts adversarial examples x adv on the proxy model f P ( x ; θ ) using white-box attack strategies and uses these malicious examples to try and fool the target model f ( x; θ).

Adversarial Examples
Given a clean sample x it is often possible to find a malicious sample x adv , such that f ( x adv ; θ) = y, and x adv is close to x according to some distance metric (either L 1 , L 2 or L ∞ norm).
For our analyses, we consider the following attacks: • Fast Gradient Sign Method (FGSM): Uses the gradient of a modified loss function to modify samples, con-strained by L ∞ norm (Goodfellow, Shlens, and Szegedy 2014).
• Virtual Adversarial Perturbations (VAP): VAP perturbs x in the direction that can most severely damage the probability that the model correctly assigns the label y (the correct label) to x (Miyato et al. 2015).
• Elastic Adversarial Perturbations (EAP): EAP is based on elastic-net regularization, which uses a mixture of L 1 and L 2 penalty functions ). These four attacks are picked so as to cover the spectrum of adversarial attacks in literature. Examples for each of these attacks are shown in Figure 1. One can also characterize attacks based on their strength, i.e. how easy or hard it is to defend against adversarial examples generated by the attacks. For example, VAP and FGSM are generally regarded as weak attacks as many defences proposed in literature have been demonstrated to be effective against VAP and FGSM attacks, as opposed to PGM which is harder to defend against.
Mathematically, all attacks can be viewed as solving the same optimization problem: an adversary tries to create a malicious sample x adv by adding a small perturbation to x, such that x and x adv are close according to some distance metric (either L 1 , L 2 or L ∞ norm), and f ( x adv ; θ) = y. Thus, the strength of an attack can be measured by how closely it can approximate the true optima of the above optimization problem.

Defences
Although many defence strategies to secure DNNs against adversaries have been proposed in the recent past, many of them can be broken when the adversary adapts the attack to take the defence into consideration. The most prominent among these strategies is adversarial training (Shaham, Yamada, and Negahban 2015) where the model is trained with adversarial samples generated by attacks along with the standard set of training examples.
Adversarial Training Adversarial training hardens the model against malicious examples by either re-training the model on an augmented set containing the training data and the adversarial examples or learning using the modified objective function: where J(θ; x, y) is the original loss function and α is a tunable hyperparameter. This defence aims to increase the model's robustness by ensuring that it predicts the same class for a clean example and its corresponding example with adversarial perturbation.
As opposed to the general practice of adding adversarial examples without replacement, we add them with replacement. This helps us achieve better generalization performance over an unseen test-set.
input: An attack A, training set D, total number of epochs N, a model with parameters θ, and learning rate η begin Intuitively, the algorithm iteratively does the following two steps • Find the optimal x adv such that f ( x adv ; θ) = y • Optimizes θ, over the worst case adversarial example

Approach
To address this challenge, we propose Adversarial Model Cascades (AMC) which can help secure models against adaptive black-box attacks. Our approach is to train a cascade of models by injecting images crafted from an already defended proxy model (or from the target model itself) to improve the robustness against adversarial attacks. In the next section, we describe the critical steps involved in our approach; namely the construction of the proxy models and the adversarial model cascades.

Proxy Models
One of the critical steps in our proposed approach is to train a proxy (or surrogate) model to mimic the target model. The strategy is to train a proxy for the target model using unlabeled examples from the proxy distribution D . The labels for these data points are obtained by observing the target model's output on these examples. Then, adversarial examples are crafted for this proxy. We expect the target model to misclassify them due to transferability between architectures (Papernot et al. 2017).
One may believe that the choice of a neural network architecture plays a vital role in the effectiveness of the proxy model and the adversary might find it hard to decide on a suitable one. However, the adversary has some partial knowledge of the oracle input (e.g., images, text) and expected output (e.g., classification) at the very least. The adversary can thus use an architecture adapted to the inputoutput relation. Adversaries can also consider performing an architecture exploration and train several substitute models before selecting the one yielding the highest misclassification. In our research, we use an architecture similar to one of those proposed by Urban et al., which they have shown to be effective in training surrogate models (via distillation) and replicating the predictive performance of the target model (Urban et al. 2016).

Adversarial Model Cascades
Inspired by the observation that adversarial examples transfer between defended models, we propose Adversarial Model Cascades(AMC) which trains a cascade of models by injecting examples crafted from a local proxy model (or the target model itself). The cascade trains a stack of models built sequentially, where each model in the cascade is more robust than the one before. The key principle of our approach is that each model in the cascade is optimized to be secure against a combination of the attacks under consideration, along with the attacks the model has encountered during the previous iteration. Knowledge from the previous model is leveraged via parameter transfer while securing the model against subsequent attacks. This technique increases the robustness of the next layer of the cascade, which ultimately yields a model which is robust to all the attacks it has been hardened against via the algorithm. A high-level overview of the AMC framework is summarized in Algorithm 2.
In the case of a white-box adversary, adversarial examples X adv are crafted for the corresponding model M i (AMC, target-hardened). For the case of an adaptive black-box adversary (AMC, local proxy), adversarial examples are crafted using an identical local proxy model P i : the proxy adversarially perturbs training data. This data is then concatenated input: Undefended model M 0 , the training set D of size N on which M 0 was trained on, a set of attacks S to harden against and the number of attacks E begin Predict using the final model in the cascade : y = arg max y M E (x); end Algorithm 2: A high-level overview of the AMC framework. The algorithm works by transferring knowledge for a specific attack and building upon it iteratively to increase robustness.
with normal training data by the target model for training. It is important to note that, for our approach to work in an adaptive black-box scenario, we need access only to the prediction interface of model M i .

Forgetting Attacks
Although AMC achieves a significant increase in robustness against many attacks, it does not guarantee an increase in robustness against all of the attacks which it has seen in the past. This condition is especially true for attacks which were seen during the initial iterations of the algorithm. To mitigate this problem, during the later iterations we also need to make the model remember the adversarial examples generated by attacks which were seen by it during its initial iterations. Thus, while constructing adversarial data per batch: instead of generating all perturbed data using the current attack, the algorithm also uses attacks it has seen so far into its run. This process implicitly weighs the attacks, as the model ends up seeing more samples from the attacks which it encounters during earlier iterations. As a result, the loss function to compute its gradient (at a given level E while running the cascade) becomes: where λ i are hyper-parameters, such that ∀i, λ i ∈ [0, 1] and T i=0 λ i = 1. With the above scheme, we observe that the resultant model remembers adversarial examples from attacks which were introduced during the initial iterations as well as recent ones, thus yielding better overall robustness.
Feature Squeezing In addition to the above, we also investigate the suitability of Feature Squeezing as a preprocessing technique in the pipeline for improving the empirical robustness of DNNs. It is important to note that Feature Squeezing as a stand-alone defence has shown to be easily bypassed (Sharma and Chen 2018). When used in conjunction with our framework, in contrast, we observe that Feature Squeezing (quantization in particular) can effectively improve the accuracy against stronger attacks.

Datasets
To measure the performance of our proposed technique, we run our experiments on three datasets standard in the computer vision/machine learning community : MNIST (Lecun and Cortes 1998), SVHN (Netzer et al. 2011) and CIFAR-10 (Krizhevsky 2009).
For training proxy models, we used the following datasets: • MNIST: we generated additional data using the technique described by Loosli et al. (Loosli, Canu, and Bottou 2007).
• SVHN: we used images from the additional set of examples available in the SVHN dataset.
To ensure a fair comparison, all the target (including AMC), as well as proxy models, were trained such that they all had accuracies within the same ballpark (Table 2).

Data Preprocessing
All pixels are scaled to [0, 1]. We employ the following split strategies for all of the three datasets: • Training data from the original dataset is used as it is for training.
• Test data from the original dataset is split into two portions. 30% of this is used as validation data by the target model while training itself. Further, 30% of the remaining 70% data, i.e., 21% of the original test data, is reserved while performing attacks. The remaining data is used as a test set for testing generalizability.
• The attacker obtains data at its level for training itself (dataset-wise, described in Section 4.1). That is, the data used by the proxy is independent of the target model.
Such a data acquisition scheme closely resembles a practical scenario, where the adversary may obtain data all by itself. The data used at all stages is balanced class-wise. Data-augmentation, similar to that described by Urban et al., is used for CIFAR10 and SVHN to help prevent overfitting (Urban et al. 2016).
AMC Hyper-parameters At the i th iteration, we perturb 80% of the data using the i th attack, while the remaining 20% data is perturbed equally, in a non-overlapping fashion, by all previous attacks. Using this strategy yields a ratio of 1.6625: 1.1625: 1.0375: 1 of the total perturbed data points seen by the model across all runs combined. The order of attacks used while running both variations of AMC is: FGSM, VAP, EAP, PGM.

Attack Hyperparameters
Based on a trade-off between attack rates and visual inspecting perturbed images, we arrived at the hyperparameters mentioned in Table 1

Evaluation Setup
We evaluate the effectiveness of our approach against adaptive black-box adversaries as follows: 1. We train two local proxy models (P & P ): (P ) is used to strengthen (or harden) the target model and (P ) is used to measure the robustness of the hardened models to adversarial attacks. These models are replicas of each other since they have the same architecture and are trained over the same dataset.

Empirical Results
We conducted multiple experiments, proving the efficiency of our method against several attacks, both in white-box and black-box setups. AMC, target-hardened uses adversarial examples generated from the target model and AMC, local proxy generates them using a local proxy. We evaluated the performance of our approach on all the three datasets listed above, for popular white-box attack algorithms such as FGSM, PGM, and compared them with both variations of our proposed framework (Section 3.2). Accuracies for the models used in our experiments are given in Table 2. Note that there is a slight drop in accuracies for the case of adversarial hardening, which has been reported widely in literature (Madry et al. 2017;Dhillon et al. 2018). Despite this fact, models trained with AMC have generalization accuracies as good as an undefended model, which is yet another advantage over plain adversarial training. The drop in accuracy for the proxy model on CIFAR-10 can be attributed to the significant difference in the distributions of data used by the proxy and target models. Parameters for the attacks we tested were decided upon after analyzing the adversarial images produced, using them. We visually inspected some of the generated examples to make sure that they are not just noise, while at the same time trying to maximize error induced in the target model * .
We compare our proposed framework (both variations mentioned in Section 3.2) with the best adversarially trained model in terms of empirical robustness for each of the attack.

White-Box Attacks
We observe that AMC, target-hardened, on an average, gives higher empirical robustness (accuracy on adversarial examples). Average robustness here refers to the robustness against all four attacks averaged together. We also observed that models obtained via adversarial hardening against only one kind of attack did not improve robustness against other attacks; whereas our models are robust against all the attacks we considered (Table 3)

Black-Box Attacks
For the adversary's proxy models, we consider adaptive proxies trained using three possible predictive interfaces.
Given an example x, the target models predictive interface returns: 1. only the most likely labelŷ, 2. only the most likely labelŷ and adds label noise to it (to approximate distillation), or 3. a vector of tuples containing class conditional probabilities p(y i |x) and the corresponding label y i .
Even in the case of black-box attacks, irrespective of the predictive interface made available by the target model, we observe trends similar to that of white-box attacks. The first type of proxy, i.e. the one that assumes only class-labels, is under the most generic scenario and thus best resembles a real-world black box setting. Results for this proxy are summarized in Table 4. We observed similar trends for the other two kinds of proxy models. Robustness of models trained with AMC (AMC,local proxy) is, on an average, higher than adversarial training.

Variations of AMC
To assert the importance of parameter transfer across iterations in our AMC algorithm, we also ran AMC with the configuration where there is no parameter transfer across the models in the cascade, i.e., (M i+1 ← θ M 0 ). As expected, we  observe the performance of the model to be dismal in the above case.
Since the algorithm processes attacks across cascades sequentially, one would expect this order to make a difference. To test this intuition, we tried two orders: sorted and reverse sorted by the power of attacks observed in the literature, i.e., FGSM, VAP, EAP, PGM and its reverse order. We observed higher robustness for models trained with the first order. Even though the ratio of attack data seen across runs is more-or-less equal, it is slightly higher for attacks seen earlier on in the algorithm. Scheduling stronger attacks first seems more intuitive, but it also adds the possibility of over-fitting to that attack.

Generalizing to an unseen Attack
Feature Squeezing Feature Squeezing (Xu, Evans, and Qi 2017) is an input pre-processing technique that re-scales inputs to get rid of high frequency, potentially adversarial, noise. It has been shown to help achieve slightly nudged accuracies on adversarial inputs. To evaluate the compatibility and advantage of Feature Squeezing on top of models trained with AMC, we calculate robustness numbers using this technique. (Table 5).
We calculated these numbers for EAP and PGM attacks; for the hyper-parameters used in earlier sections (Table 1) as well as higher parameters (doubled 'nb iter' and 'eps' for PGM, 'binary steps' and 'max iterations' for EAP) to  achieve higher error rates. It is important to note that Feature Squeezing as a stand-alone defence has shown to be easily bypassed (Sharma and Chen 2018). When used in conjunction with our framework, in contrast, we observe that Feature Squeezing (quantization in particular) can effectively improve the accuracy against stronger attacks, samples from which weren't previously seen by the model.

Leave one attack out validation
To study the capability of models trained via AMC to generalize, we test it against an unseen attack: we use n − 1 attacks for running AMC, and the n th one for performing attacks. We experimented using FGSM as the unseen attack in a white-box setting. For all datasets, models trained with AMC outperformed undefended models significantly in terms of robustness (Table 6). These numbers are, on an average, better than all of the defence methods (apart from hardening against FGSM, since in that case the attack is known before-hand) in Table 3.

Discussion
As we observed in our experiments, there is no defence technique that works against all adversarial attacks which are ef-fective for complex datasets such as SVHN and CIFAR10; hardening a model for a specific attack does not necessarily lead to an increase in robustness against future attacks of that kind. Adversarial training, as we observed, increased robustness only against the attack it is hardened against and did not substantially increase robustness against other attacks significantly.
The key contributions of the AMC framework proposed in the paper are as follows : 1. AMC provides robustness against all the attacks it is hardened against, making it an all-in-one defence mechanism against several attacks.
• In the case of white-box attacks, on an average AMC provides an absolute increase in robustness of 6.225% for MNIST, 5.075% for SVHN and 2.65% for CI-FAR10, in comparison to adversarial hardening. • In the case of black-box attacks, on an average AMC provides an absolute increase in robustness of 0.45% for MNIST, 6.25% for SVHN and 2.7% for CIFAR10, in comparison to local-proxy based adversarial hardening.
2. Even though Feature Squeezing has been shown to be bypassed, when used in conjunction with our framework, we observe that Feature Squeezing (quantization in particular) can effectively improve the robustness against both attacks seen by the model during the training (on an average, an absolute increase of 2.65% for MNIST, 8.9% for SVHN and 8.8% for CIFAR10) and stronger attacks which weren't previously seen by the model (on an average, an absolute increase of 68.9% for MNIST, 56.25% for SVHN and 58.2% for CIFAR10) on top of AMC.
3. There is no overhead at inference time. Thus, services can deploy versions of the models hardened with AMC without any compromise on latency.
4. It is easy to incorporate into already trained models. As the intermediate step in building the model cascades involves fine-tuning , it is much faster than existing defensive methods which require training from scratch for every attack.  (Table 2).
6. We also observe that the performance of AMC for unseen attacks is comparable to models hardened against those specific attacks (Table 6). An interesting direction for future would be to evaluate the performance of AMC over a larger set of unseen attacks.
To the best of our knowledge, ours is the first attempt to provide an end to end pipeline to improve robustness against multiple adversarial attacks simultaneously; in both whitebox and black-box settings.