{'fraction_non_alphanumeric': 0.03714515069692768, 'fraction_numerical': 0.008976136612420638, 'mean_word_length': 5.098798397863819, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
BARReL: Bottleneck Attention for Adversarial Robustness in Vision-Based Reinforcement Learning

Robustness to adversarial perturbations has been explored in many areas of computer vision. This robustness is particularly relevant in vision-based reinforcement learning, as the actions of autonomous agents might be safety-critic or impactful in the real world. We investigate the susceptibility of vision-based reinforcement learning agents to gradient-based adversarial attacks and evaluate a potential defense. We observe that Bottleneck Attention Modules (BAM) included in CNN architectures can act as potential tools to increase robustness against adversarial attacks. We show how learned attention maps can be used to recover activations of a convolutional layer by restricting the spatial activations to salient regions. Across a number of RL environments, BAM-enhanced architectures show increased robustness during inference. Finally, we discuss potential future research directions.


Introduction
Visual-based reinforcement learning (RL) is a sub-field of reinforcement learning research that uses an image as input for the decision-making process. Prominent examples include video games like Atari [1,8] or robotics applications. Learning from raw pixels is a promising approach because it is generally applicable and requires little feature engineering, but it can be challenging in practice. While RL has profited from advances in deep learning architectures for computer vision, it has inherited its susceptibility to adversarial attacks [6,10]. In fact, trained RL agents are often very sensitive to the model to the quality of the visual input, i.e., they are easily corrupted due to environ-* Equal contribution mental/equipment factors, like different lighting conditions, shadowing, camera quality/damage, or by adversarial intentions of the third parties, like adversarial attacks. The robustness of such systems is crucial for potential future realworld use, especially in safety-critical applications. In this work, we (1) propose a conceptually simple, general, yet effective inference-time method for defense against adversarial attacks. (2) In initial experiments, we demonstrate the effectiveness of the approach. Finally, we (3) discuss potential future research directions and extensions.

Related work
Adversarial attacks in Reinforcement Learning There are a number of previous works that tackle the problem of adversarial attacks on reinforcement learning agents. In RL, the goal of adversarial attacks is to deteriorate the policy's performance. Similar to the supervised settings, attacks can be classified into whiteand black-box attacks [5]. Beyond attacks found in supervised settings, which are comparable to attacking the state/observation space of an agent [2,10], there exist additional attacks such as poisoning the environment dynamics or reward function [5]. In this work, we focus on white-box state space attacks, i.e., attacks on the input observations in which the attacker has access to the underlying model. Attention methods in Computer Vision Different regions of an image are not equally important for predictions. Identifying the most important part can be done via attention mechanisms. There are various approaches to model attention in computer vision [4]. In this work, we utilize Bottleneck Attention Modules (BAM) [9] which applies both spatial and channel-wise attention. Following common practice, we utilize frame stacking of a small sequence of proceeding game frames, which means that here channel-wise Figure 1. We specify all the tensors on top of picture of the tensor with blue italic font and operations with green font providing legend in top-right corner. The architecture of the encircled Bottleneck Attention Module can be found in Park et al. [9]. The dotted red line indicates the elements used in the "recovery" mechanism based on attribution maps. attention enables temporal attention. Defenses against adversarial attack Existing defense strategies include methods like adversarial training and game-theoretic approaches to defend against inference-time attacks or robust learning against training-time attacks [5]. In this work, we combine adversarial training with a novel defense strategy to defend against inference-time attacks.

Method
For our approach, we draw inspiration from selective attention [14]: only a part of visual input is necessary for decision making, i.e., a significant part of an input image contains non-relevant information. Gradient-based adversarial attacks modify both relevant and non-relevant parts of visual input. The entirety of the perturbation can corrupt the predicted output action distribution of an RL agent and consequently cause potentially erroneous behavior. Thus, our goal is to mitigate the effect of these attacks by removing non-relevant parts of the input, which can dramatically reduce the magnitude of the perturbation. We utilize Bottleneck Attention Module [9], which has been introduced as a simple and non-invasive extension to existing CNN architectures, to learn attention maps of the most relevant spatial and temporal features for vision-based RL. We use learned attention maps to "recover" the output of a convolutional layer perturbed by an adversarial attack. In effect, we use the attention maps to only retain relevant parts of the input.

Implementation Details
Training We use Proximal Policy Optimization [13], with the implementation and default training hyperparameters from the StableBaseline3 library [11]. We train all models on three Atari environments, which are the part of a well-established benchmark used to compare RL methods [1], namely: Breakout-v4, SpaceInvaders-v4, MsPacman-v4. We apply the common pre-processing routing for observation; namely, transformation to greyscale, resizing, a frame-skipping of 4 frames, as well as stacking of the 4 last frames to encode temporal dynamics. As a baseline architecture (Nature-CNN), we use a 3-layer CNN, followed by two fully-connected layers and a softmax that outputs a categorical probability distribution over a set of possible actions similar to [8]. We extend this architecture by embedding a BAM-layer [9] between the first and second convolutional layer (BAM-CNN). These architectures contain 2.246.416 parameters (Nature-CNN) and 2.248.409 parameters (BAM-CNN), respectively. Figure 1 shows the used architecture. To have a performance baseline, we trained both Nature-CNN and BAM-CNN architectures on the default environments (denoted without any additional suffix in Tab. 2). We utilize adversarial training by applying Projected Gradient Descend (PGD) attacks [7] using Foolbox framework [12] with an = 0.1 and get two additional training regimes: Nature-CNN-Adv. and BAM-CNN-Adv.. In RL, the quality of training is directly dependent on the learned policy, thus applying attacks too aggressively, e.g., at every step, might stop an agent from learning altogether. We found that attacking only every 10th frame had no significant impact on training performance see Tab. 2), but in turn only gave moderate robustness to adversarial attacks (Tab. 3 shows that adversarial attacks are successful with approx. 100% probability). However, our experiments showed that adversarial training seems to be necessary to learn appropriate attention maps. When trained on the default environment, the learned attention maps were generally not suited to support the recovery of activations. Inference Performance of the RL agents at the inference stage with respect to cumulative reward, averaged over 10 episodes, for all of the environments is presented in Tab. 2 that empirically confirms that both: (1) introducing of BAM layer to architecture and (2) incorporation of adversarial samples during training do not lead to dropping of the performance. Input: ψ θ Output:π(a|s A ) that is close to π * (a|s) f (l−1) ← ψ [1:l−1] θ (s A ) For notation clarification see 1

Recovery algorithm
inal states s are created by an attack algorithm A, where controlling the severity of the perturbation. The goal of the proposed algorithm is to recover a policyπ(a|s A ) that is close to the original policy π * (a|s) induced by ψ θ . The policies' similarity metrics are discussed in Sec. 4. We present the pseudo-code of the inference-time defense algorithm in Alg. 1. Our analysis shows that this particular reinforcement learning scenario produces highly polarized attention maps (i.e., very high contrast between high-and low-attention areas). Therefore, the learned BAM maps have a strong notion of what is important for decision-making. As a result, applying element-wise (Hadamard) multiplication allows us to "prune" part of the input that is not significant for the decision-making but still affected by adversarial perturbations.  Table 2. Cumulative reward results of baselines: Nature-CNN, BAM-CNN, Nature-CNN-Adv., BAM-CNN-Adv. for Breakout-v4, SpaceInvaders-v4, MsPacman-v4 averaged over 10 episodes with standard deviations. For the adv. attacks, each frame is attacked with an = 0.05.. We contrast these scores with the reward achieved by our proposed method (see Tab. 1).  Table 3. Attack recovery results with different values for Breakout-v4, SpaceInvaders-v4, MsPacman-v4, averaged over 10 games each. The used architecture was trained on partly attacked data during training (with every 10th frame perturbed), which only resulted in marginal adversarial robustness. This adversarial training by itself is not able to effectively defend against adversarial attacks: As is visible in the first line, adversarial attacks manage to switch the selected action in most cases, even for small . Applying the presented recovery technique can decrease the impact of attacks up to a certain level.

Breakout-v4 MsPacman-v4 SpaceInvaders-v4
tially reversed attack (Reversed-TOP-2) when the recovery leads to the predicted action matching either the first or second choice of the original action. (c) The percentage of partially reversed attack (Reversed-ANY) when the recovered action is different from the attacked action (or already the original action assigned maximum probability). We also report the (d) percentage of successful PGD attacks (% Successful Attacks), i.e., how often the attack is able to switch the maximum probability action. The results for different across different environments can be found in Tab. 3. Figure 2 shows qualitative visual results of the "recovery" technique. The adversarial attack targets the entire frame, which is clearly visible in the activations. The recovered activation shows how a significant part of the applied perturbations is removed from the input image. As a result, the output action distribution after the "recovery" procedure is closer to the original action distribution (see Tab. 3). For example, in Breakout-v4 for moderate = {0.01, 0.05}, the original action is restored with 57.78% and 35.51% probability, respectively (out of 4 total actions). For MsPacman-v4, for approx. 30% of states the original action is restored (out of 9 possible actions). This recovery is noticeable in the final performance of the trained agents attacked with adversarial observations. We see improvements that scale with the success rate of the defense.

Discussion & Future work
A great strength of the presented method is it's simplicity and low application cost. The additional BAM-layer and clean-up process during inference only adds marginal complexity in terms of model parameters and inference time. BAM itself is suitable for any type of CNN architecture, but we might explore other types of attention (e.g., selfattention) for other architectures like vision transformers [3]. Another potential benefit is the use of selective atten-tion to increase robustness against other types of distractors like background images. While the initial results showcase the potential of the method, the performance of agents is still worse in the attacked environments. It is, therefore, necessary to explore how to improve the method further. Another shortcoming is the fact, that we applied the presented defense in a grey-box setting, in which the attacker is not aware of the inference procedure. To further investigate robustness, we should explore adaptive attack procedures [15] with the attacker having full knowledge of the attack. Furthermore, a comparison to other defense methods [5] is warranted. Besides the aforementioned improved evaluation, we plan to extend our method in follow-up work: (a) Investigating its utility for core computer vision problems and different backbones that do not share characteristics of reinforcement learning and the investigated environments. (b) Investigate different "recovery" procedures on feature maps, e.g., using different operations besides element-wise multiplication. (c) Use multiple BAM attention layers in a network and apply the clean-up-procedure multiple times, which might increase robustness, (d) Investigate the possibility of application of fewer amount of attacks during training. (e) Explore different types of adversarial attack methods. (f) Extend the current method to vector-based RL and continuous action spaces.

Conclusion
In this paper, we presented a simple adversarial defense method for deep RL based on Bottleneck Attention. This type of selective attention could be a general tool to restrict the effect on non-targeted adversarial perturbations. We presented initial promising results that demonstrate the potential of the method to recover activations perturbed by adversarial attacks. In future work, we plan to apply a more thorough evaluation and study of potential improvements.