{'fraction_non_alphanumeric': 0.03352616355508809, 'fraction_numerical': 0.020970286615829607, 'mean_word_length': 4.921759439470611, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Impact of Attention on Adversarial Robustness of Image Classification Models

Adversarial attacks against deep learning models have gained significant attention and recent works have proposed explanations for the existence of adversarial examples and techniques to defend the models against these attacks. Attention in computer vision has been used to incorporate focused learning of important features and has led to improved accuracy. Recently, models with attention mechanisms have been proposed to enhance adversarial robustness. Following this context, this work aims at a general understanding of the impact of attention on adversarial robustness. This work presents a comparative study of adversarial robustness of non-attention and attention based image classification models trained on CIFAR-10, CIFAR-100 and Fashion MNIST datasets under the popular white box and black box attacks. The experimental results show that the robustness of attention based models may be dependent on the datasets used i.e. the number of classes involved in the classification. In contrast to the datasets with less number of classes, attention based models are observed to show better robustness towards classification.


I. INTRODUCTION
Recently, the vulnerability of deep learning models to adversarial attacks has been recognized. Adversarial attacks in the context of image classification involve fooling an image classifier by generating an image with small perturbation from the original image which results in misclassification of the image. Previous works in the field have proposed a variety of explanations for the existence of adversarial examples including high-dimensional geometry of data manifold [1] and computational limitations of learning algorithms [2]. Ilyas et al. [3] emphasize that adversarial robustness is a human-centric phenomenon and vulnerability to adversarial examples exists because models learn to treat non-robust features and robust features with equal importance.
Attention plays a huge role in the human visual system. Humans use a sequence of partial glimpses and selectively focus on important parts instead of processing the entire scene at once. Inspired by the attention mechanism in the human visual system, several methods have been proposed to incorporate attention in convolution neural network (CNN) based image classification tasks which have also led to improvement in the accuracy of the classifiers e.g. residual attention network [4], squeeze and excitation networks [5], convolutional block attention module [6], etc. [7].
Following this context, the present work focuses on studying the impact of learning salient features through attention mechanisms on the adversarial robustness of a model using the image classification problem. The major contribution of the present research work is as follows: • Comparative analysis of the robustness of attention based (using mixed attention) and non-attention based models under the white box and transfer based black box attacks. • Exploring the dependence of adversarial robustness of attention and non-attention based models on datasets used. • While recent works have proposed attention based techniques to improve adversarial robustness, our study focuses on understanding how attention impacts robustness which would give more insights on the existence of adversarial examples and how defense strategies could be developed. • To the best of our knowledge, this is also the first attempt to analyze the adversarial robustness of image classification models with mixed attention.
The rest paper is divided into several sections, where Section 2 describes the literature survey of the works describing robustness of the classification models, followed by Section 3 presenting the proposed framework to simulate white box and black box attacks on classification models with different datasets. Section 4 presents the the simulation results of the attacks on the models. Finally, concluding remarks are presented in Section 5.

II. RELATED WORK
In order to defend against adversarial attacks, extensive research has been conducted to develop defense strategies and build robust models. Some of them are robust training, input transformation, randomization, model ensemble, and certified defenses.
In robust training, the classifier is made robust against small noises internally. Adversarial training [8]- [10] is a popular technique based on robust training which augments training data by adversarial examples. Several methods based on robust training using losses or regularizations have been proposed. These include regularization on input gradient [11], perturbation based regularization [12] and variants on network Lipschitz constant [13]. Another technique is to transform the inputs before feeding them into the classifier. Total variance minimization [14], autoencoder based denoising [15], using generative models to project adversarial examples onto the data distribution [16] are some of the input transformation based strategies. Adding randomness to the input [17], [18] or the model [19] have also been proposed to enhance robustness. Another defense strategy is to construct an ensemble of models and aggregate the output of each model in the ensemble [20], [21]. There are a lot of works based on certified defenses which are guaranteed to be robust against adversarial perturbations under some attack models [22]- [25].
Several recent works have proposed models with attention based mechanisms to enhance robustness. Zoran et al. [26] use a visual attention component guided by a recurrent top down approach to improve the adversarial robustness, whereas Liu et al. [27] propose a nonlinear attention module for feature prioritization to improve adversarial robustness. In similar work, Zhao et al. [28] show that self-attention networks may have benefits in terms of adversarial robustness and later, Vaishnavi et al. [29] use foreground attention masks i.e separate the background from the foreground and mask the background before training and classification. Their results show an increase in adversarial robustness on using foreground masks. Goodman et al. [30] develop a method called attention and adversarial logit pairing that encourages both attention map and logit for pairs of clean examples and their adversarial counterparts to be similar. In this work, we attempt to get a general understanding of the impact of attention in image classification models using mixed attention under different kinds of attacks.

III. PROPOSED WORK
Based on the discussion in Section 1, we conduct experiments to understand if learning of salient features through attention mechanisms translates to adversarial robustness through better identification of underlying features in adversarial images. We compare the adversarial robustness of an image classifier with and without attention under the white box and black box attacks. Fig 1 shows a schematic representation of the proposed framework in which the attacks are simulated.
We have considered Resnet-50 [11] for image classification and used Convolutional Block Attention Module (CBAM) to

A. Attacks
The attacks used in this work can be classified into white box and black box. In white box attacks, an attacker has complete access to the target model including the model parameters, architecture, training method and training data, and exploits this information to craft an attack. In black box attacks, the attacker has no access to model parameters and uses a different model or a query based strategy to perform an attack.
1) White box attack: Popular white box attacks include fast gradient sign method [8] (FGSM), projected gradient descent [9] (PGD), deepfool [31], C&W [32]. In this work, FGSM and PGD have been used to evaluate and compare the performance of attention and non-attention models under white box attack scenarios.
FGSM is a single step gradient based approach where an adversarial example is created by maximizing the loss function L(x adv , y) wrt to the input x and then adding the sign of output gradient to x in order to produce x adv (adversarial example) and is represented in Eq. 1. In our analysis, the accuracy of the target models is studied for various values of epsilon in the range of 0 to 0.3. PGD uses the projected gradient descent to iteratively create adversarial examples. The perturbed data in step t + 1 can be represented in Eq. 2.
Here, α is the step size and π x+S denotes projecting perturbations into set S. In all our experiments, we use PGD bounded under l inf norm with = 0.2 and α = 0.01. The attack accuracy is evaluated for different numbers of iterations.   Table I for classification models over multiple datasets.

B. Datasets used
Publicly available datasets: CIFAR-10 [33], CIFAR-100 [33] and Fashion MNIST [34] have been used in our work. CIFAR-10 and CIFAR-100 consist of 60k colour images of size 32x32 the training set consisting of 50k images and the test set of 10k images. CIFAR-10 has 10 classes of 6k images each and CIFAR-100 has 100 classes of 600 images each. Fashion MNIST dataset consists of grayscale images of 10 fashion classes. The size of images is 28x28 and consists of 60k training images and 10k testing images. Table II shows the performance of the non-attention model and attention based model trained on CIFAR-10, CIFAR-100 and FashionMNIST datasets under FGSM and PGD attacks with different values of perturbation and number of iterations respectively.

A. White box
1) CIFAR-10: A comparison of white box FGSM attack for different perturbation values shown in Fig. 2(a) suggests that CBAM+Resnet-50 is attacked to a much larger extent as compared to Resnet-50 throughout the range of epsilon values from 0.05 to 0.3, where the accuracy of CBAM+Resnet-50 even drops to ∼ 0%. The white box PGD attack in Fig. 2(b) also represents towards better robustness of the non-attention model (Resnet-50) since it takes a larger number of iterations as compared to the attention model (CBAM+Resnet-50) for complete attack i.e. 100% error. 2) CIFAR-100: Unlike the observation for white box attacks on the CIFAR-10 dataset, the attention based model shows better robustness for the CIFAR-100 dataset. White box FGSM attack ( Fig. 3(a) shows slightly better accuracy for CBAM+Resnet-50 but the PGD attack ( Fig. 3(b) convincingly describes that CBAM+Resnet-50 is much more robust as compared to Resnet-50 since it isn't attacked completely even in more than 20 iterations while Resnet-50 is completely attacked in less than 5 iterations. 3) Fashion MNIST: Fig. 4(a) shows that Resnet-50 shows better adversarial robustness upto a perturbation value of 0.1.
For epsilon values greater than 0.1, both CBAM+Resnet-50 and Resnet-50 have approximately equal accuracy under the FGSM attack. Fig. 4(b) depicts the better robustness of Resnet-50 since it takes a larger number of iterations as compared to CBAM+Resnet-50 for complete attack.  Table III shows the performance of the non-attention model and attention based model trained on CIFAR-10, CIFAR-100 and FashionMNIST datasets under transfer based FGSM and PGD attacks under the 3 scenarios of source and target chosen which has been described in Table 1. The performance of the models have been observed for different values of perturbation in case of FGSM and different number of iterations for PGD.

1) CIFAR10:
For the FGSM attack, Resnet-50 showed better transferability leading to a better attack on CBAM+Resnet-50 as compared to an attack on Resnet-50 by examples generated by CBAM+Resnet-50 ( Fig. 5(a)). Under the FGSM transfer attack with VGG-16 as the source, the attention and non-attention model both showed approximately equal accuracy under attack (Fig. 5(b)). Under the transfer based PGD attack, Resnet-50 was comparatively more robust to attack against adversarial examples generated by CBAM+Resnet-50 (Fig. 5(c)). Resnet-50 thus showed better transferability for the attack against attention based model. However, under transfer based PGD with VGG-16 as the source, CBAM+Resnet-50 shows better robustness as compared to Resnet-50 (Fig. 5(d)).
2) CIFAR-100: For the FGSM attack, Resnet-50 showed better transferability leading to a better attack on CBAM+Resnet-50 as compared to an attack on Resnet-50 by examples generated by CBAM+Resnet-50 ( Fig. 6(a)). Under the FGSM transfer attack with VGG-16 as the  source, the attention and non-attention model both showed approximately equal accuracy under attack ( Fig. 6(b)). Under the transfer based PGD attack, Resnet-50 was comparatively more robust to attack against adversarial examples generated by CBAM+Resnet-50 (Fig. 6(c)). Resnet-50 thus showed better transferability for the attack against attention based model. However, under transfer based PGD with VGG-16 as the source, CBAM+Resnet-50 shows better robustness as compared to Resnet-50 ( Fig. 6(d)).

3) Fashion MNIST:
Resnet-50 is much more robust to transfer based FGSM attack from adversarial examples generated by CBAM+Resnet-50. Comparison between source and target accuracy in Fig. 7(a) shows that Resnet-50 has good transferability since the accuracy of CBAM+Resnet-50 under attack differs from Resnet-50 as a source by a small margin. The transferability of CBAM+Resnet-50 under FGSM attack is much poorer which can be observed through the large difference in accuracy between CBAM+Resmet-50 (source) and Resnet-50 (target) (Fig. 7(b)). Under the FGSM transfer based attack with VGG-16 as a source, there's not much difference in the robustness of the non-attention and attention model. The attention based model (CBAM+Resnet-50) is slightly more robust for perturbation values greater than 0.  Fig. 7(c)). A similar result is observed for PGD attack with VGG-16 as source where Resnet-50 shows better robustness against adversarial examples generated by VGG-16 ( Fig. 7(d)).

V. CONCLUSION
While it was expected that attention based models would be more robust under white box attacks and transfer based black box attack and would also show better transferability, the results obtained for CIFAR-10 and Fashion MNIST dataset strongly favour the robustness of the non-attention model i.e. Resnet-50. However, for the CIFAR-100 dataset, the attention based model i.e. CBAM+Resnet-50 was comparatively much more robust than the non-attention model Resnet-50 under white box attacks while the difference in results between the two models was very low in the transfer based black box attacks. These results suggest that the impact of attention on the robustness of image classification models could be dependent on the datasets used i.e. the number of classes in the datasets. Similarity in results of CIFAR-10 and Fashion MNIST and deviation of CIFAR-100 from these results could be due to fewer number of classes i.e. 10 classes in comparison with CIFAR-100 dataset which has 100 classes. The results obtained in this work is a step closer towards a generalized understanding of the impact of attention on the robustness of image classification models. Experiments with more datasets with different number of classes and models with other kinds of attention could be some directions for future work.