{'fraction_non_alphanumeric': 0.04820193637621024, 'fraction_numerical': 0.019778699861687413, 'mean_word_length': 4.416104868913858, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ensemble of Models Trained by Key-based Transformed Images for Adversarially Robust Defense Against Black-box Attacks

We propose a voting ensemble of models trained by using block-wise transformed images with secret keys for an adversarially robust defense. Key-based adversarial defenses were demonstrated to outperform state-of-the-art defenses against gradient-based (white-box) attacks. However, the key-based defenses are not effective enough against gradient-free (black-box) attacks without requiring any secret keys. Accordingly, we aim to enhance robustness against black-box attacks by using a voting ensemble of models. In the proposed ensemble, a number of models are trained by using images transformed with different keys and block sizes, and then a voting ensemble is applied to the models. In image classification experiments, the proposed defense is demonstrated to defend state-of-the-art attacks. The proposed defense achieves a clean accuracy of 95.56 % and an attack success rate of less than 9 % under attacks with a noise distance of 8/255 on the CIFAR-10 dataset.


INTRODUCTION
Intentionally perturbed data points known as adversarial examples are indistinguishable from clean data points, but they cause deep neural networks (DNNs) erroneous predictions [1,2]. The existence of such adversarial examples has been an alarming concern since DNNs are to be deployed in security-critical applications such as autonomous vehicles, healthcare, and finance. Therefore, a lot of effort has been put towards adversarial robustness.
Researchers have proposed various adversarial attacks and defenses. However, most of the conventional defenses either reduce classification accuracy significantly (e.g., adversarial training [3]) or are completely broken [4,5]. The research on adversarial robustness has entered into an "arms race" between attacks and defenses [4][5][6][7]. Recent works [8][9][10] proposed defense methods with a secret key from a cryptographic point of view. The main idea of these methods is to embed a secret key into the model structure with minimal impact on model performance. Assuming the key stays secret, an attacker cannot compute any useful gradients on the model, which will render the existing gradient-based adversarial attack such as Projected Gradient Descent (PGD) [3] ineffective. However, an attacker may perform gradient-free attacks (i.e., black-box attacks) without the secret key when the output of a model is available.
Therefore, in this work, we adapt the defense method in [9] and propose a voting ensemble to defend against blackbox adversarial examples. In the proposed method, a number of models are trained by using images transformed with different keys and block sizes. One of the models is used as a front-end model and outputs the probability of prediction to users, and a final class label is determined by using majority votes of all the models in the ensemble. We make the following contributions in this paper.
• We propose an ensemble of models protected by using secret keys for the first time. • We conduct three state-of-the-art black-box attacks and present the effectiveness of the proposed defense. In image classification experiments, the proposed defense is confirmed not only to outperform the previous key-based adversarial defenses [8,9], but also to outperform state-of-theart defenses [11,12].

RELATED WORK 2.1. Adversarial Attacks
Mainly, there are two types of attacks: white-box (complete knowledge of the model and its training data) and blackbox (no knowledge).
White-box: Given an input image x and a classifier f (·), an adversarial example x is generated such that f (x ) = y, where y is a true class. This is done by minimizing the perturbation δ, or by maximizing the loss function, Usually, a typical threat model is bounded by an p norm such that ∆ = {δ : δ p ≤ } for some perturbation distance > 0. Some of the most popular gradient-based (white-box) attacks are Fast Gradient Sign Method (FGSM) [2], Projected Gradient Descent (PGD) [3], Carlini and Wagner (CW) [13], etc.  Black-box: There are also gradient-free (black-box) methods that estimate gradients such as [6,14,15]. Another recent black-box attack, NATTACK, learns a probability distribution centered around the input such that a sample drawn from that distribution is likely an adversarial example [16]. A recent score-based black-box attack based on a randomized search scheme is even more powerful than gradient-based white-box attacks [17].

Adversarial Defenses
The goal of a defense method is to make a model that is accurate not only for clean input but also for adversarial examples. Many different approaches that try to achieve this goal, such as certified and provable defenses [18,19], adversarial training [3,11,20], preprocessing techniques [21,22], and detection algorithms [23,24]. All of the defense methods either reduce classification accuracy or are completely broken [4,5] and detection methods can be bypassed [7].
Recently, a new line of adversarial defense methods was introduced by using a secret key as in cryptography [8,9]. The work by [8] utilized pixel shuffling in a pixel-wise manner to input images by standard random permutation. However, their work dropped accuracy drastically even on dataset like CIFAR-10. The work by [9] proposed pixel shuffling in a block-wise manner to maintain high classification accuracy. Although their work was confirmed to be effective against white-box attacks when the key is secret, black-box attacks that do not require the secret key to compute gradients are possible. In this work, we adapt the work by [9] and propose a voting ensemble method to prevent from black-box attacks.

Overview
Previous key-based adversarial defenses [8][9][10] work well against white-box attacks by holding a secret key at a provider as shown in Fig. 1. White-box attacks require correct gradients that cannot be computed without the secret key. There- fore, the previous works [8][9][10] are effective against whitebox attacks. However, gradient-free (black-box) attacks that do not need to access the secret key can be still applied to the key-based defenses. Therefore, the proposed defense aims to extend the key-based defenses to defend against black-box attacks.
The proposed method is an ensemble of models trained by using images transformed with different keys and block sizes. An overview of the proposed defense is shown in Fig. 2. We first select block size M n ∈ {M 1 , . . . , M N }, and then N models are trained by using images transformed with the selected M n and a key K n . One of the models in the ensemble is a front-end model (i.e., public-facing model) that outputs the probability of the prediction. A final class label is determined by voting prediction results from all models.

Block-wise Image Transformation with Secret Keys
Models in the proposed ensemble are trained by using block-wise transformed images with secret keys as well as in [9]. The following are steps for transforming images, where c, w and h denote the number of channels, width, and height of an image tensor x ∈ [0, 1] c×w×h , and N is the number of models.
to obtain a shuffled vector, b (i,j) i.e., Integrate the shuffled vectors to form a shuffled image tensor x n ∈ [0, 1] c×w×h .
The above steps are carried out for every model f n , n ∈ {1, . . . , N }. Each model (f n ) in the ensemble is trained by using transformed images (x n ). Model f 1 is a front-end (public-facing) model which outputs a class probability to users.

Selection of Block Size
The previous works [9,10] confirmed that selecting a larger block size for block-wise image transformation provides better robustness against attacks, but a lower classification accuracy. In contrast, using a smaller block size achieves a higher classification accuracy, but lower robustness. From these results, we chose two block sizes for training models: a block size of M 1 = 16 for the frontend (public-facing) model and M (2 to N ) = 2 for the other models. The effectiveness of this selection will be demonstrated in an experiment.

Ensemble
Ensemble methods have been used as a technique applied to the improvement of model predictions in general [25]. There are many forms of the ensemble: voting, bagging, boosting, and stacking. In this work, we utilize voting where the output class label is determined by the majority of votes. The proposed voting ensemble consists of two steps: 1. Each model computes an output probability vector z n = [z n (1), . . . , z n (L)], where L is the number of class labels. 2. The final prediction y final is determined by using the majority of the votes from the models, i.e., y final = mode(argmax(z 1 ), . . . , argmax(z N )). (4)

Threat Model
To evaluate a defense method, precisely defining threat models is necessary. A threat model includes a set of assumptions such as an adversary's goals, knowledge, and capabilities [26]. We consider untargeted attacks where the goal of the adversary is to reduce the classification accuracy of the model. In [9], it is confirmed that white-box attacks such as PGD [3] are not effective when the key is secret because correct gradients cannot be computed without the key. Therefore, in this work, we focus on black-box attacks, specifically, SPSA [6], NATTACK [16], and SQUARE [17] attacks. The adversary performs untargeted evasion attacks (i.e., test time attacks) in which small changes under ∞ metric change the true class of the input. The adversary's capability is to modify the test image where the noise distance is 8/255.

Set-up
We used the CIFAR-10 [27] dataset with a batch size of 128 and live augmentation (random cropping with padding of 4 and random horizontal flip) on the training set. CIFAR-10 consists of 60,000 color images (dimension of 32 × 32 × 3) with 10 classes (6000 images for each class) where 50,000 images are for training and 10,000 for testing.
We utilized deep residual networks [28] with 18 layers (ResNet18) for the CIFAR-10 dataset and trained for 200 epochs with efficient training techniques from the DAWN-Bench top submissions: cyclic learning rates and mixedprecision training. The parameters of the stochastic gradient descent (SGD) optimizer were a momentum of 0.9, weight decay of 0.0005, and a maximum learning rate of 0.2.
Three black-box attacks: SPSA [6], NATTACK [16], SQUARE [17] were deployed to evaluate the proposed defense. NATTACK was configured with a population size of 300, a sigma value of 0.1, a learning rate of 0.02, and 100 iterations. SPSA was set up with a delta value of 0.01, a learning rate of 0.01, a batch size of 256, and 100 maximum iterations. We used 2000 queries for SQUARE attack. The noise distance was 8/255 for all attacks.
To evaluate the proposed defense, we used accuracy (ACC) over the whole test dataset (10,000 images) and attack success rate (ASR) over randomly selected 1000 images that are correctly classified by the ensemble. The goal of a defense method is to make a model that has both a high ACC value and a low ASR one. Table 1 summarizes the performances of the proposed defense in terms of ACC and ASR values under the use of various block sizes used in image transformation, where M (1 to N ) indicates M 1 = M 2 = · · · = M N , and M (2 to N ) means M 2 = M 3 = · · · = M N , and N is the number of models. When the same block size as M (1 to N ) was applied to all models, the proposed ensemble with M (1 to N ) = 2 achieved almost the same accuracy as the baseline model (without any defense) in terms of clean accuracy. However, as demonstrated in [9,10], using a smaller block size for the block-wise transformation achieves a higher classification accuracy on clean images, but the robustness against adversarial examples is lower (a higher ASR value). Therefore, to achieve both a higher ACC value on clean images and a low ASR even under attacks, we selected two block sizes for training models such as M 1 = 16 and M (2 to N ) = 2. From Table 1, the proposed defense with two block sizes allowed us not only to maintain high ACC values but also to achieve lower ASR ones. In particular, The ensemble with M 1 = 16 and M (2 to N ) = 2 outperformed the other ensembles under the attacks.

Selection of the Number of Models
The experiment in Table 1 was carried out with an ensemble size of N = 9. In contrast, Table 2 shows the performance of the proposed defense under the use of variable ensemble sizes (i.e., N ∈ {3, 5, 7, 9}) in terms of ACC values for both clean images and adversarial examples. From the table, the proposed defense was demonstrated to be able to maintain high ACC values even under variable ensemble sizes. In particular, the ensemble with N = 9 achieved the highest clean accuracy.

Comparison with State-of-the-art Defenses
We compared the proposed ensemble with state-of-theart defenses: fast adversarial training (Fast AT) [11], feature scattering approach (FS) [12], key-based standard random permutation (SRP) [8] and block-wise shuffling (Key-based M = 4) [9] in terms of clean ACC and ASR. The ASR of Fast AT for NATTACK was ≈ 40 % and that of the other two attacks was ≈ 32 %. The ASR of FS was lower than that of Fast AT (20.48 % for NATTACK and ≈ 30 % for the other two attacks). The models, Key-based (M = 4) and SRP were completely defeated by black-box attacks (i.e., the ARS was almost 100 %). In contrast, the ASR of the proposed ensemble was lower than 9 % for SPSA and less than 2 % for the other two attacks. Therefore, the proposed defense outperformed the state-of-the-art defenses under all the three black-box attacks.

CONCLUSION
In this paper, we proposed a new adversarial defense that is a voting ensemble of key-protected models to overcome black-box attacks for the first time. Models in the ensemble were trained by using images transformed with different keys. We evaluated the proposed defense under three different state-of-the-art black box attacks. The results showed that clean accuracy of the proposed defense achieved more than 95 % and the attack success rate was less than 9 % for the three black-box attacks with a noise distance of 8/255 on the CIFAR-10 dataset. Comparing with state-of-the-art defenses, the proposed ensemble outperformed both the previous keybased defenses and the conventional adversarial defenses. As for future work, we shall optimize the ensemble to further improve the performance of the proposed defense.