{'fraction_non_alphanumeric': 0.03620661019764159, 'fraction_numerical': 0.009715994020926755, 'mean_word_length': 4.648452157598499, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Understanding Pixel Vulnerability under Adversarial Attacks for Images

Deep neural network image classifiers are reported to be susceptible to adversarial evasion attacks, which use carefully crafted images created to mislead a classifier. Recently, various kinds of adversarial attack methods have been proposed, most of which focus on adding small perturbations to all of the pixels of a real image. We find that a considerable amount of the perturbations on an image generated by some widely-used attacks may contribute little in attacking a classifier. However, they usually result in a more easily detectable adversarial image by both humans and adversarial attack detection algorithms. Therefore, it is important to impose the perturbations on the most vulnerable pixels of an image that can change the predictions of classifiers more readily. With the pixel vulnerability, given an existing attack, we can make its adversarial images more realistic and less detectable with fewer perturbations but keep its attack performance the same. Moreover, the discovered vulnerability assists to get a better understanding of the weakness of deep classifiers. Derived from the information-theoretic perspective, we propose a probabilistic approach for automatically finding the pixel vulnerability of an image, which is compatible with and improves over many existing adversarial attacks.


Introduction
Recently, Deep Neural Networks (DNNs) have enjoyed great success in many application areas such as computer vision and natural language processing. Nevertheless, DNNs have been demonstrated to be vulnerable to adversarial attacks, which are data samples carefully crafted to be misclassified by DNN classifiers [6,19,12]. For example, in image classification, an adversarial example may be perceived as a legitimate data sample in a ground-truth class but misleads a DNN classifier to predict it into a maliciously-chosen target class or any class different from the ground truth.
The most popular way to generate adversarial examples is by adding small perturbations/noise to the pixels of a real image, e.g. in [6,18,3,25,1]. For many existing attacks, a large amount of the pixels 1 of a real image are perturbed to generate an adversarial example, with the same (e.g., Fast Gradient Sign Method (FGSM) [6]) or different magnitude (e.g., Projected Gradient Descent (PGD) [17]), according to the gradient of the classifier loss for each pixel. Our research has found that a large proportion of the generated perturbations of a successful adversarial example contribute little to the attack against the classifier. By removing those ineffective perturbations for an existing attack method, we are to able attack with significantly fewer perturbations and keep its attack performance almost the same. In this way, existing attacks can be refined to generate more realistic adversarial examples, which are obviously less easily detected by humans. Moreover, we can empirically show the adversarial examples generated by these refined attacks can also significantly lower the accuracy of adversarial detectors [5,26,15,20]. To identify effective perturbations on a real image for an attack method, we need to find out the "vulnerability" of all the pixels of the image, as perturbations are more effective if they are imposed on more vulnerable pixels This is a challenging task as the pixel vulnerability is specific to the real image, the classifier, and the attack. In Figure 1, we depict a demonstration on MNIST [13] using the PGD [17] attack on a convolutional neural network classifier 2 . Specifically, given the real images, our proposed method in this paper can automatically detect each pixel's level of vulnerability, shown in (b) and (f). Guided by the detected vulnerability, we impose the PGD perturbations on the most vulnerable pixels and ignore the others. In this way, we can obtain 98% of the attack performance of the original PGD by only perturbing the top 30% most vulnerable pixels. It is also noteworthy that the vulnerability map can greatly help us to get a more intuitive and deeper understanding of the weak points of a classifier under adversarial attacks. For example, it is difficult to tell how a PGD attack changes digit "3" to "9" by looking at its perturbations. However, the vulnerability map intuitively shows that it is the perturbations on the "break of the upper ellipse" of digit "3" that are most likely to result in the misclassification. This concurs with our visual perception.
In this paper, we propose a probabilistic approach to detect pixel vulnerability as shown in Figure 1. we propose a probabilistic approach derived from an information-theoretic perspective. Specifically, given a real image, we first generate its adversarial example by an arbitrary attack, which we call the "source attack". Next, a neural network is used to generate a vulnerability map for a natural image, which is trained by an algorithm derived from an information-theoretic perspective. We then construct a stochastic procedure that selects a subset of the pixels of an image and only imposes the perturbations of the source attack on the selected pixels. In this way, based a source attack, we develop a new attack that aims to attack as well as the source attack with fewer pixels perturbed.
The contributions of this paper can be summarised as follows: i) We comprehensively study the problem of detecting an image's pixel vulnerability under an adversarial attack, which is important for developing better attack methods and more robust defence algorithms. To our knowledge, this has not been comprehensively addressed in previous research. ii) We introduce a new principled probabilistic approach that can find pixel vulnerability in an end-to-end manner. We believe that our adaption of probabilistic approaches to adversarial learning is novel and inspiring. iii) We propose a new adversarial attack framework guided by the learned pixel vulnerability, which is compatible with most of the existing adversarial attack techniques and improves over them with significantly fewer perturbations resulting in reduced detectability without sacrificing attack performance. iv) Our proposed approach helps develop an intuitive and deep understanding of the weaknesses of deep classifiers, which can inspire the development of more robust defence algorithms.
2 Background and proposed approach

Background
Suppose that a normal image and its ground-truth label are denoted by X andŷ ∈ {1, · · · , K}, respectively, where K is the number of unique image labels. We consider a pretrained neural network classifier f , which takes X as input and outputs a probability vector over K labels, i.e., y = f (X).
Usually for a well-trained classifier, we haveŷ = arg max y. An adversarial attack is to find perturbations η (with the same dimensions as X) and add them to X to give the adversarial example, X adv , which is expected to mislead the classifier to fail to identify the true label. Importantly, the adversarial image shall be inside an -ball around X, i.e., η < where · can be the L ∞ norm in accordance with [17,1] or other norms. Finding the value of η can be formulated as the following optimisation problem: max η < (f (X + η),ŷ), where (f (X),ŷ) denotes the loss function (e.g., cross-entropy loss) of the classifier f given input X and its true labelŷ.

Problem definition
Here we assume an image X consists of H × W pixels, where each pixel x ij can have either one channel (greyscale images) or multiple channels (colour images). Given X and a classifier f , we first apply a "source attack", s, resulting in the source adversarial example X s : X s = s(X), whose predicted label distribution is y s = f (X s ). In general, s can be an arbitrary adversarial attack that adds perturbations, as long as it can attack the classifier well and its perturbations fall into the -ball. Moreover, s(X) can be viewed as a stochastic process for many existing attacks, for example, PGD [17] attacks with random initialisation in the -ball. For most attacks, all of the pixels of X are perturbed, but usually with different magnitudes. In this paper, we demonstrate that the pixels in an image have different vulnerability and the perturbations on the most vulnerable pixels contribute most in changing the the classifier's decision.
For X, we introduce a vulnerability map Θ ∈ R H×W , generated from a neural network g parameterised by α taking X as input: Θ = g α (X). Specifically, θ ij indicates the vulnerability of pixel x ij . Based on Θ, we develop a stochastic procedure E (detailed in Section 2.3) that selects a subset of the pixels to perturb. Specifically, a draw from E is a binary matrix: Z ∼ E(Θ) ∈ {0, 1} H×W , where z ij = 1 indicates pixel x ij is selected and vice versa. Given X s and Z, we construct a new adversarial example by imposing the source attack's perturbations only on the selected pixels: where indicates the element-wise product and η = X s − X is the matrix of perturbations from the source attack. Although Z is binary, it does not mean that a pixel's vulnerability is binary. As E is a stochastic process, a pixel with lower vulnerability will be less frequently selected in multiple draws of E. Finally, we can describe our paper's research problem as: Given X and X s , finding the optimal Θ to generate X adv , which attacks the classifier f as well as possible. Here we propose a probabilistic learning algorithm for Θ derived from an information-theoretic perspective. Before detailing the proposed learning method, we first view our model as a graphical model with random variables, demonstrated in Figure 2. Specifically, we note: i) y is the predicted label distribution of X and y|X ∝ f (X); ii) X s is generated by a source attack from X; iii) y s is the predicted label distribution of X s and y|X s ∝ f (X s ); iv) Θ is generated by g α (X); v) Z is drawn from E parameterised by Θ; and vi) X adv is generated given Z and X s by Eq. (1).

Learning
In practice, most of the published source attacks are able to achieve impressive attack performances. For example, PGD with 20 iterations can reduce the accuracy of the state-of-the-art classifier on the CIFAR10 [11] dataset from 92.7% to 0.8% [17]. Therefore, it is reasonable that we learn Θ by narrowing the margin of the attack performances between X adv and X s . As mutual information is a measure of the dependence between two random variables and captures how much knowledge of one random variable reduces the uncertainty about the other, we propose a learning objective that maximises the mutual information between y s and X adv , as follows: As X s comes from the source attack, it already satisfies the constraint of the -ball. Therefore, there is no need to consider this constraint when learning Z. It is worth noting that Eq. (2) is not directly implementable, so we consider the following derivation: In the derivation of Eq. (3), we leverage the fact: as shown in Figure 2. According to Figure 2, we also have: , the learning process can be described as: For one input image X, we first sample X s and Z; then we accordingly sample X adv ; X s and X adv are fed into the classifier to get their predicted label distributions; finally, we minimise the margin between the two label distributions. Recall that X adv uses a subset of the perturbations of X s . If it can attack as well as X s , the selected pixels in X adv shall be more vulnerable than the others, which indicates a good vulnerability map has been learned.
Now we introduce the details of the stochastic process E that generates Z. Inspired by [4], among all the HW pixels of an image, we maximally select M ∈ {1, · · · , HW } pixels to impose on the perturbations from the source attack. Next, we introduce a matrix version of categorical distribution parameterised by Θ, a draw of which picks one pixels among the HW pixels: U ∼ Categorical(Θ).
Here U ∈ {0, 1} H×W is a one-hot binary matrix, where the entry selected by the categorical distribution is turned on and all the others are zeros 3 . To maximally select M pixels, we draw U for M times and choose the pixels that are selected at least once, formulated as follows: where U <m> denotes the output matrix of the m th draw and max m U <m> denotes the element-wise maximisation, i.e., z ij = max m u <m> ij .
Recall that we aim to train the neural network parameter α to optimise Eq. (2). To avoid backpropagation through the categorical random variables, we utilise the Concrete distribution [9,16] to approximate the categorical distribution:Ũ ∼ Concrete(log Θ), whereŨ ∈ R H×W + is the continuous relaxation of the one-hot matrix U in Eq. (4). Specifically, we have: where γ ij is from a Gumbel(0, 1) distribution: γ ij = − log(− log ν ij ), ν ij ∼ Uniform(0, 1), and τ is the "temperature". With the Concrete distribution, we have: whereZ is an approximation to Z for backpropogation in the learning phase.
Finally, we elaborate on the training process in Algorithm 1. After the neural network g is trained, given an input image X, we can first get Θ, then select pixels to perturb according to Θ, and finally use Eq. (1) to generate the adversarial image X adv .

Related work
input :Classifier f , Source attack s, Input image collection X , Maximum number of vulnerable pixels M output :Parameter of the neural network α while Not converged do Sample a batch of input images X B ; forall X ∈ X B do Generate source adversarial image X s = s(X); for m = 1 to M do DrawŨ <m> by Eq. (6); end CalculateZ by Eq. (7); Generate final adversarial image X adv = X +Z (X s − X); Calculate learning loss L by Eq. (2); Compute noisy gradients of L in terms of α; end Average noisy gradients of batch samples; Update α by stochastic gradient steps and backpropagation; end Algorithm 1: Learning algorithm Here we first review the related work in the domain of adversarial attacks for deep learning. As adversarial attack is a vast and increasingly popular research topic, we limit our discussions to perturbation-based attacks. Besides FGSM [6]) and PGD [17]), there are many other perturbation-based attacks, such as BIM [12], DeepFool [18] and CW [3]. Most of the above methods are non-trainable attacks, meaning that they conduct an optimisation process for each individual image. Several trainable methods have been proposed such as Adversarial Transformation Networks (ATNs) [2], AdvGAN [25], and Rob-GAN [14]. Those trainable methods usually consist of a generator that is learned with training images and used to directly generate adversarial examples. Our proposed approach falls into the category of trainable attacks as well. It is noteworthy that any perturbation-based attack can serve as the source attack in our model, as we only require the output from a source attack as our model's input.
To the best of our knowledge, the closest attack to our approach is the Jacobian-based Saliency Map Attack (JSMA) [22], which uses a saliency map computed according to the gradients of the classifier loss and then perturbs those pixels with high saliency. Several extensions and variations have been proposed to JSMA, including those in [3,24]. Although JSMA's high-level idea on perturbing a subset of pixels is related to ours, the aims and mechanisms are fundamentally different. First of all, JSMA is an independent attack, while ours can be used to analyse and improve over many existing attacks. Moreover, JSMA is a non-trainable attack, while ours is trainable. Our approach is more efficient in performing attacks on test images.

Experiments
In this section, we conduct experiments to evaluate the performance of our method, named PVAdv (Pixel Vulnerability Adversary), on attacking image classifiers. It is noteworthy that our method is not designed to be an alternative attack that competes with existing approaches. Instead, any existing perturbation based attacks can serve as PVAdv's source attack and PVAdv is able to discover the vulnerable pixels of an image for the source attack then keep its attack performance with fewer perturbations . Here we focus on untargeted white-box attacks, though the attacks discussed in this paper (including the proposed one) can be adapted to targeted or black-box settings.

Evaluation metrics
We first report a classifier's accuracy on the adversarial images generated from the test images of a dataset (AdvAcc), to test if PVAdv keeps the source attack's performance with fewer perturbations.
To verify whether our attack is less easily detected than its source attack, following [5], we build an adversarial detector to predict whether an image is natural or adversarial. The detector consists of a logistic regression with three features computed from the output logits of the classifier: classifier confidence, kernel density (K-density), and the entropy of normalized non-maximal elements (non-ME). The former two features are introduced in [5] and the latter is from [20]. For an attack method, we split the test set of a dataset in half, then use the images of the first half and their adversarial counterparts to train the detector, and report the Area Under Curve (DetAUC) on the second half.
Finally, inspired by [27], we protect a naive classifier with the adversarial detector. Suppose that both natural and adversarial images are sent to the classifier to classify but the classifier does not know which ones are adversarial. This is a highly practical case of adversarial attacks in the real world. Before an image goes to the classifier, we use the adversarial detector to check whether it is natural or adversarial 4 . If the image is detected to be natural, regardless of whether it is actually natural or adversarial, the classifier classifies it. Any images that are detected to be adversarial by the detector are rejected for classification. The detector-protected classifier is considered to be successful on an image if: A) the image is adversarial and gets rejected; B) the image is accepted and successfully classified. In Case B, if the detector mistakenly lets an adversarial image in, the classifier can hardly successfully label it, given the strong performance of adversarial attacks. We report the adversarial accuracy for the detector-protected classifier (Protected-AdvAcc) computed by (#A + #B)/N , where #A and #B are the numbers of images in Case A and B, respectively, and N is the total number of images. Interestingly, a strong attack method is not guaranteed to attack the protected classifier well, as it can be easily detected and rejected by the detector. Therefore, attack methods need to be "smart", i.e., attacking on more vulnerable pixels with fewer perturbations. Protected-AdvAcc is reported on the second half of the test images of each dataset, similar to DetAUC. We note that from an attacker's perspective, the lower the three metrics are, the better.

Experimental settings
Here we consider the MNIST [13] and CIFAR10 [11] datasets, where the pixel values of the images were normalised between 0 and 1. Settings of classifiers: We use the naturally-pretrained classifiers in the Madry Challenges [17]: a Convolutional Neural Network (CNN)-based model for MNIST and a ResNet-based one [8] respepctively, which are widely-used in robust machine learning. Settings of source attacks: We select two popular adversarial attacks as the source attacks for PVAdv: FGSM [6] and PGD [17]. We used the default settings of the two source attacks implemented in CleverHans [21]. Note that the source attack is not limited to the above two and can be an arbitrary perturbation-based attack. We set to 0.3 for MNIST and 0.03 for CIFAR10, respectively, which are the standard settings. Settings of PVAdv: For the neural network g, we leverage an autoencoderlike architecture similar to the one of [23], which is further based on the code of WGAN-GP [7]. Specifically, the "encoder" encodes an input image into a 128-dimensional latent representation with a CNN and the "decoder" decodes from the latent representation to the vulnerability map of the input image with a deconvolutional network. We set M = round(βHW ), where β ∈ (0, 1) is a hyperparameter introduced to control the proportion of perturbed pixels. The temperature of the Concrete distribution is set to 0.5, following [4]. We implement PVAdv in Python with TensorFlow, trained by Adam [10] with learning rate 0.0001 and batch size 100. We train the model for 500 iterations maximally and terminate the training if the loss stops dropping for 10 continuous iterations. Settings of other baselines: Besides the two source attacks that serve as natural baselines for PVAdv, we compare with JSMA [22], which also attacks with a subset of pixels. We use the implementation in CleverHans, where the target label is selected randomly and the number of perturbed pixels is set to the same values as M in PVAdv. Interestingly, JSMA can also be used as our source attack.

Results
We show the results on MNIST and CIFAR10 in Table 1. First, it can be observed both FGSM and PGD achieve good results for attacking the classifiers on MNIST and CIFAR10. However, the the DetAUC for both of them is obviously large, for example, it is over 93% on both MNIST and CIFAR10 for PGD, which indicates that their attacks are easily detected by the detection algorithm. Once detected, their attacks can hardly harm the performance of the protected classifier. For example, in CIFAR10, PGD achieves 0.03 accuracy on the original classifier, but only gets 0.892 accuracy   on the protected classifier, as it gets a very high DetAUC of 0.981. It is interesting to see that PGD attacks better than FGSM against the unprotected classifier, but its performance is usually worse than FGSM on the protected classifier, due the high detectability of PGD.
With FGSM and PGD as our source attacks, the proposed PVAdv is able to attack as well as those source attacks with much fewer perturbations, e.g., PVAdv PGD with β = 0.5 (i.e., only 50% of the pixels are perturbed) on MNIST and CIFAR10. Accordingly, our method's DetAUC is much lower than the source attacks, leading to significantly lower values in Protected-AdvAcc. For example, PVAdv PGD with β = 0.5 achieves 0.672 against the protected classifier for CIFAR10 compared with 0.892 for the original PGD, without sacrificing the attacking performance on the unprotected classifier (i.e., AdvAcc of PGD and PVAdv PGD with β = 0.5 is 0.0030 and 0.0032, respectively).  between X and X adv divided by the number of perturbed pixels and report the mean L 2 over all the testing images. The results of PVAdv and PVAdv reverse with β = 0.5 are listed in Table 3, showing that the perturbation magnitude of the 50% pixels selected by PVAdv is similar to that of the pixels selected by PVAdv reverse , but PVAdv achieves much better AdvAcc. So the effectiveness of PVAdv is not due to the perturbation magnitudes but positions indicated by the vulnerability map. The running time of performing attacks on 100 sampled images is reported in Table 4. We run all the methods in TensorFlow on a PC with a NVIDIA TI-TAN RTX GPU. Once trained, our proposed PVAdv attacks very efficiently (e.g., at least 10X on MNIST and 100X on CIFAR10 faster than JSMA).

Conclusion
To identify the vulnerable pixels of an image that can actually challenge classifiers, we have proposed a novel probabilistic approach called PVAdv, derived from the information-theoretic perspective. Given an arbitrary source attack, PVAdv is able to discover vulnerability maps of images and keep the source attack's performance with much less perturbations. This significantly reduces the detectability the source attack and leads to better attack performance on detector-protected classifiers. The vulnerability maps also handily assist the understandings to the source attack and provide insights for developing more effective defence methods. Future research is on the defence side, aiming to improve the classifier robustness by better protecting the vulnerable pixels.