{'fraction_non_alphanumeric': 0.03955967605358578, 'fraction_numerical': 0.007534142475628026, 'mean_word_length': 4.643671118735449, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Characterizing Attacks on Deep Reinforcement Learning

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.


INTRODUCTION
With recent progress of DRL in various applications, such as computer games [12,21,22], autonomous navigation [9,26] and robotics [17], the safety and robustness of DRL models are becoming a major concern, especially on real world robotics tasks [17,30,35]. Recently, adversarial attacks that have imposed challenges to general deep neural network (DNN) models [14,18,33,[37][38][39][40] started to challenge the robustness of DRL models. However, the adversarial attacks on DRL models can be different from attacks on DNN models: DRL models focus on sequential decision-making problems while DNN models mostly work on individual prediction problems with no temporal continuity. Existing white-box attacks on DRL models assume almost full access to the victim policy [16]. Some black-box attacks assume partial knowledge of the victim policy [13,16] but attack each observation individually. These blackbox attacks can be computationally intensive especially on tasks with high dimensional inputs. Moreover, most of these attacks have only been evaluated in simulated environments. It remains unclear of their effects on real world DRL models.
In this work, we seek to expand our understanding of the vulnerabilities of DRL systems. To achieve this goal, we propose a set of realistic attacks with improvement on the computational efficiency. To make less assumptions on the victim model, we focus on black-box attack. Based on the components of MDP (shown in Figure 1), we categorize attacks into different types, including attacks on observation space and on environment dynamics. To improve the attack efficiency on multiple high dimensional inputs, we propose to reduce the amount of computation by utilizing the sequential nature of MDP tasks. Finally, to validate the feasibility of adversarial attacks on real world DRL tasks, we perform real world physical attack on a real robot control task. We summarize the proposed attacks on DRL models and our contribution as follows.
Advanced black-box attacks. An attacker may not have access to the DRL system's internals, making white-box attacks infeasible in this case. For DNN models, black-box attacks that take advantage of query access to the victim model have emerged [6]. Based on this progress, we adapt and improve this method for attacking DRL systems and demonstrate the effectiveness. Specifically, starting from the finite difference (FD) based attack [6], we propose an improvement, named adaptive sampling FD (SFD), that reduces the amount of computation by adaptively sampling the input dimensions for gradient estimation. We provide a theoretic analysis of SFD method and prove its efficiency and estimation error bound.
High throughput perturbation. The adaptive SFD method is still inefficient since the attacker would need to generate adversarial examples on each individual frame. To make further improvement, we propose an online sequential FD attack based on the fact that DRL tasks have temporal continuity, where temporal consecutive frames tend to correlate with each other. We hypothesize that the attacks generated on a small group of selected frames can be applied globally to other similar and temporally close frames. When limiting the number of frames to be selected, it is important to find the best set of frames for generating such attacks. We observe that not all action decisions (frames) are critical within a trajectory, and hypothesize that attacks that are generated based on a small group of important and critical frames can be more effective. Based on this, we propose an optimal frame selection approach to select the most important frames to generate the attack. We provide mathematical analysis of this approach.
Perturbations on components other than the observation. As shown in Figure 1, the observation space is not the only component in DRL systems. Besides, environment dynamic is another important component in the DRL systems. Thus, we propose another attacks that perturb the environment transition dynamics by changing physical properties of the environment rather than changing the input observation to the victim model.
Physical attacks. While we have shown the digital adversarial examples on DRL systems, it is still unclear about the possibility to generate physical adversarial examples to attack the physical DRL system. We bridge this gap by using a printed adversarial patch and a toy robot visual navigation task. We show that the overall endto-end attack is effective even under effects such as manufacturing inaccuracy and varied viewing angles.
We conduct extensive experiments on the above proposed attacks and compare them with existing white-box (which could be viewed as the performance upper bound) and black-box attacks both in simulation and on a real robot. We show that it is feasible to attack real-world DRL systems with our proposed approaches.

RELATED WORK
Adversarial attacks on machine learning models. Our attacks draw inspirations from previously proposed attacks. Goodfellow et al. [14] describes the fast gradient sign method (FGSM) of generating adversarial perturbations in a white-box setting. Carlini and Wagner [7] describe additional methods based on optimization, which results in smaller perturbations. Moosavi-Dezfooli et al. [24] demonstrates a way to generate a "universal" perturbation that is effective on multiple inputs. Xiao et al. [39] generates adversarial examples in 3D world by changing the shape and texture information respectively. Evtimov et al. [11] shows that adversarial examples can be robust to natural lighting conditions and viewing angles using real world examples. Furthermore, black-box attacks without providing victim model's training algorithms are also proposed for general machine learning models [8,31].
Adversarial attacks on DRL. DRL methods train a policy that maps state observations to action decisions. Examples include Deep Q-Learning (DQN) [23] for discrete control, and Deep Deterministic Policy Gradient (DDPG) [19] for continuous control. Proximal Policy Optimization (PPO) [34], and Soft Actor-Critic (SAC) [15] are also proposed recently. We select DQN and DDPG as our target victim algorithms, but our attacks can apply to other RL methods.
Recently, Huang et al. demonstrates an attack that uses FGSM to perturb observation frames in a DRL setting [16]. However, the white-box setting in this work requires knowing the full victim model and the preferred action. They also propose a black-box attack method based on transferability, where the surrogate models are trained to obtain attacks and the generated attacks are then applied on the victim models. We build on FD-based attacks that do not rely on transferability. Lin et al. [20] designs an algorithm to achieve targeted attack for DRL models, and they propose a method to select optimal frames for attacks based on the preference of the policy on the best action over the worst action. We provide related theoretical proofs to demonstrate the optimality of frame selections. Behzadan and Munir [4] propose a black-box attack method that trains another DQN network to minimize the expected return using FGSM. Gleave et al. [13] proposes to train another agent to interact and modify the environment so as to indirectly attack the victim model. The black-box attacks in these related works are all evaluated on simulated environments. We provide real world experiments validating the effectiveness of our proposed Table 1: Summary of the adversarial attacks on DRL systems, categorized based on our proposal. The name reflects the category of the attack method. For example, obs-fgsm-wb means attack on observation using fast gradient sign method based white-box attack and obs-fgsm-bb means attack on observation using fast gradient sign method based black-box attack. The attack methods we proposed are highlighted using bold text. "Arch.," "Param.," and "Query" indicate whether the attack requires knowledge of the policy network's architecture, parameters and whether it needs to query the policy network.  [28] propose to use candidate inference attack to infer possible dynamics used for training a candidate policy, posing potential privacy-related risk to deep RL models. Robust RL via adversarial training. Safety and generalization in various robotics and autonomous driving applications have drawn significant attention for training robust models [25,27,32]. Knowing how DRL models can be attacked is beneficial for training robust DRL agent. Pinto et al. proposes to train a RL agent to provide adversarial attack during training so that the agent can be robust against dynamics variations [32]. However, since they manually selected the perturbations on environment dynamics, the attack provided in their work may not be able to generalize to broader RL systems. Additionally, their method relies on an accurate modeling of the environment dynamics, which may not be available for real world tasks such as robotics systems.

THREAT MODEL ON DRL
Our victim models are trained by interacting with environments that are Markov Decision Processes (MDPs), which include several components: the state space S, the action space A, the transition dynamics T and the reward function R. The goal of the DRL model is to learn a policy so as to maximize the agent's future expected where is a discount factor. In this work, we provide methods for adversarial attack for trained DRL policies, including discrete control and continuous control methods. We select two representative algorithms: DQN [23] for discrete control and DDPG [19] for continuous control.
We aim to attack well-trained DRL models without accessing the victim model's parameters and only querying the victim model to get model output. The goal of the attacker is to minimize the agent's future expected return. We do not assume these attackers have full control over the agent nor the robotics system. They are weaker than, for example, an attacker that could generally compromise the robot's software. To avoid trivial detection, an adversary needs to constrain the magnitude of perturbation. In this work, we bound the inf norm of the added perturbation during the evaluation of the digital attacks. For real world physical attacks, we follow the common settings in the literature [2,11].

ADVERSARIAL ATTACKS ON DRL
In this section we develop several concrete attacks to improve the attack feasibility and efficiency. We first introduce some baseline attacks and then describe our new attacks in detail. Table 1 summarizes these attacks, where we categorize them based on their attack components (attack observation or transition dynamics), attacker's knowledge, the computational efficiency of the attack (real-time), whether the attack requires physically changing the environment (physical), and whether the attack is based on temporally dependency of consecutive frames (independent or sequential).

Baseline Attacks
We discuss both white-box and black-box baseline attack methods.
White-box attacks. In this setting, we assume that the attacker can access the agent's policy network ( | ) where refers to the action and refers to the state. Huang et al. [16] has previously introduced one attack in this category that applies the FGSM method to generate white-box perturbation purely on observations. We reproduce this experiment with our obs-fgsm-wb attack. This attack's application scenario is when we know the policy network's architecture and parameters.
Black-box attacks. There are different scenarios depending on the attacker's knowledge. In one scenario, the attacker does not have any information about the model architecture, or parameters, and it can't query the model either. In this case, the attacker can perform a "transferability" based attack by attacking a surrogate model (of which it has complete knowledge) and then transfer the perturbation to the victim model. Huang et al. [16] introduced a black-box variant of the FGSM attack using transferability, which we denote as obs-fgsm-bb. Their experiments assume that the attacker has access to the same environment, knows the training algorithm of the targeted agent, and uses the same algorithm to train the victim model.

Advanced Black-Box Attacks
In an alternative black-box setting, the attacker has access to query the model, obtaining the model's outputs on given inputs (while still not knowing the model architecture or parameters). This setting represents a realistic real-world deployment, for example where an executable copy of the agent is shipped to customers, or where an online service is offered. In this setting, we propose more advanced black-box attacks, which take advantage of this query access.
Black-box finite difference (FD) based attack. Baseline blackbox attack obs-fgsm-bb requires retraining a surrogate policy. Previous work [6] applies the finite difference (FD) method in attacking classification models. We extend the FD method to DRL systems in obs-fd-bb which doesn't require retraining a new policy. This attack works in the setting where we don't have the policy network's architecture or parameters and the training algorithms, but can query the network. FD based attack on DRL uses FD to estimate gradient with respect to the input observations. It then generates perturbations on the input observations by using the estimated gradient. The key step in FD is to estimate the gradient. Denote the loss function as and state input as s ∈ R . The canonical basis vector e is defined as an dimension vector with 1 only in the -th component and 0 otherwise. The FD method estimates gradients via the following equation where is a parameter to control estimation accuracy. The loss function depends on the actual RL algorithm we use, and since we have the model output, we can select an action with a minimal value. Define the state-action value function as ( , ), then given a state , we can obtain a bad target action from the model output as = arg min ( , ). Then we define to induce the model to select that bad action as follows. Denote the actor of the RL algorithm as ; the loss function for continuous control is: (s) = ∥ (s) − ∥ 2 2 , and the loss function for discrete control is, (s) = CELoss( (s), ), where CELoss() is the cross-entropy loss for classification. In the DQN setting, there is no actor, but we can define an action probability distribution as .
For dimensional input, the finite difference method would require 2 queries to obtain the estimation, which is computationally intensive for high dimensional inputs such as images . Therefore, we propose a sampling technique to mitigate this computational cost.
Adaptive sampling based FD (SFD). Many deep learning models extract features from inputs patch-wise and have sparse activation map [3]. Based on the this observation, we propose a method for estimating gradients that exploits this spatial structure. In this method, we first estimate the gradient with respect to some randomly sampled pixels, then iteratively, we identify pixels where the gradient has a high magnitude and estimate the gradient with respect to surrounding pixels.
Given a function (·; ) : R → R 1 , where is the model parameter (we omit this for conciseness below), our goal is to estimate the gradient of with respect to an input ∈ R : ∇ˆ( ). We define the nontrivial dimension of the gradient of at as { ∈ {1, 2, · · · , }; |∇ ( )| ≥ }, i.e., the dimensions with gradient absolute value greater or equal to a threshold value > 0.
To estimate nontrivial dimension of the gradient, first, we randomly sample dimensions from {1, · · · , }, to get a set of dimensions = { 1 , 2 , · · · , }, and use FD to estimate the gradients for dimensions in . Then we select a set of dimensions ′ = { ∈ ; |∇ ( ; )| ≥ }, and use FD to estimate the gradients of the neighbors (a set ′′ ) of dimensions in ′ , if these gradients haven't been estimated (for dimension within a dimensional vector, the neighbor dimension is defined as dimension + 1 if it exists). Then we select dimensions with absolute gradients no less than from ′′ and find their neighbors to estimate gradients. We repeat this process for multiple iterations. By exploring the sparse large gradients this way, we can adaptively sample dimensions to estimate gradients, which can significantly reduce the number of queries. We give the full attack algorithm of obs-sfd-bb. in our appendix. We denote obs-s[n]fd-bb as the attack of obs-sfd-bb with iterations.
To better understand the benefits of SFD, here we provide an analysis of this algorithm and estimate the amount of nontrivial dimension of the gradient that can be estimated using our method in Lemma 1. The basic idea of this lemma is to prove that by using SFD, we can sample more of the nontrivial dimensions of the gradient than by using random sampling. We also provide an error bound for the estimated gradient with SFD in Theorem 2.
Definition 4.1 (Neighbor Dimension's Gradient). ∀ , ∈ {1, 2, · · · , } and = +1, we define the neighbor dimension's gradient as ∇ ( ) = ∇ = +1 ( ). Note that = +1 is equivalent to = −1, and to be general we choose the first one to obtain the definition. Definition 4.2 (Non-trivial Gradient Dimension). Given a positive gradient threshold , an input data instance ∈ R , and a loss function : R → R 1 , for any dimension ∈ {1, · · · , }, if |∇ ( )| ≥ , then we define this gradient as non-trivial gradient and the corresponding dimension as non-trivial gradient dimension. On the other hand, if |∇ ( )| < , then we define this gradient as trivial gradient and the corresponding dimension as trivial gradient dimension. Definition 4.3 (Gradient Sampling Probability). Given a selected threshold > 0 in Algorithm SFD, for any ∈ R , define the non-trivial gradient sampling probability as, where represents the set of dimensions selected by algorithm . Therefore, the gradient sampling probability of SFD and random sampling are SFD ( ) and random ( ) respectively. Some further definitions on neighbor gradient distribution probability are as following: • If |∇ ( )| ≥ + , then define • If |∇ ( )| ∈ [ , + ], then define Based on the above assumption that these distribution 1 , 2 , 3 and are defined over all possible dimensions in one image (over ) and these distribution works throughout the entire gradient estimation iteration process, we have the following lemma. Lemma 1. We make the following assumptions on : For dimension whose gradient |∇ ( )| ∈ [ , + ], the probability that the gradient magnitude of its neighborhood pixel is in [0, ] is 1 . We conclude, as long as 1 The intuitive understanding of this lemma is that when nontrivial gradients (of magnitude no less than ) dimensions are spatially concentrated ( 1 is small, then 1 < 1 − random ( )), our method will be more sample efficient than random sample method. We provide the proof of this Lemma in our appendix. Next we give another theorem about the upper bound for the gradient estimation error and include the proof for this theorem in our appendix.
Theorem 2. Suppose we sample all nontrivial dimensions of the gradient and estimate the gradient with perturbation strength , the estimation error of the gradients is upper bounded by the following inequality, for constant > 0, = =1 1(|∇ ( )| ≥ ), and ∇ˆ( ) is the estimated gradient of with respect to .

High Throughput Attacks
A DRL system operates on a sequence of consecutive frames. To develop a useful attack method against real-time DRL systems, we must consider the computational costs. Therefore, in this section, we propose a method for generating perturbations efficiently: an online sequential attack.
In a DRL setting, consecutive observations are not i.i.d.-instead, they are highly correlated and sometimes the consecutive observations do not change too much. It's then possible to perform an attack with less computation than performing the attack independently on each state. Considering real-world cases, for example, an autonomous robot would take a real-time video as input to help make decisions, an attacker is motivated to generate perturbations only based on previous states and apply it to future states, which we refer to as an online sequential attack. We hypothesize that a perturbation generated this way is effective on subsequent states. Therefore, we propose online sequential attacks obs-seq-fgsmwb in whitebox setting and obs-seq-fd-bb, obs-seq-sfd-bb in blackbox setting. In these attacks, we first collect observation frames and generate a single perturbation using the averaged gradient on these frames (or estimated gradients using FD or SFD, in the case of obs-seq-fd-bb and obs-seq-sfd-bb). Then, we apply that perturbation to all subsequent frames. We denote them as obsseq[Fk]-fgsm-wb, obs-seq[Fk]-fd-wb,obs-seq[Fk]-sfd-wb.
Next, instead of using all the frames, we further improve the above attack by finding the the set of frames that are important and using the gradients from those frames to perform attack. With this, we hope to maintain attack effectiveness while reducing the number of queries needed. We propose to select a subset of frames within the first frames based on the variance of their values. Then, in all subsequent frames, the attack applies a perturbation generated from the averaged gradient. We select an optimal set of important frames with highest value variance to generate the perturbations. We denote them as obs-seq[Lk]-fgsm-wb, obs-seq[Lk]-sfd-wb, obs-seq[Lk]-sfd-wb. We give a proof in Corollary 3 below for why attacking these important frames is more effective to reduce the overall expected return. We include the proof in our appendix. Corollary 3. Let the state and state-action value be ( ) and ( , ) respectively for a policy with time horizon . We con- where ( 1 =ˆ1 ) means the observation at time 1 is changed from 1 toˆ1 .

Attacks on Components Other than the Observation
Besides attacking the observation space, the attacker can have access to the testing environments of the victim model. Therefore, potentially the attacker can modify the environment such as changing the physical properties of the environment to perform the attack.
In this case, the observations of the victim model are not perturbed but the environment transition dynamics will be perturbed. RL based attacks on environment dynamics. In this case, the attacker will perturb the environment transition model (dynamics). Such transition model is usually non-differentiable with respect to the policy. Therefore, we propose a novel reinforcement learning based method to attack environment dynamics. We describe a targeted attack (in which the agent will fail in a specific way, e.g. a Hopper turn over, or a self driving car drive off the road and hit obstacles.) where the attacker changes the environment dynamics (e.g. by changing the mass of the car). The algorithm is as follows.
Define the environment dynamics as M, the agent's policy as , the agent's state at step following the current policy under current dynamics as , and define a mapping from , M to : ∼ ( | , M, 0 ), which outputs the state at time step : given initial state 0 , policy , and environment dynamics M. The task of attacking environment dynamics is to find another dynamics M ′ such that the agent will reach a target state ′ at step : We consider the following two algorithms for generating this attack: First, Random dynamics search. A naive way to find the target dynamics, which we demonstrate in env-rand-bb, is to use random search. Specifically, we randomly propose a new dynamics and see whether, under this dynamics, the agent will reach ′ . This method works in the setting where we don't need to have access to the policy network's architecture and parameters, but just need to query the network. Second, RL based adversarial dynamics search. We design a more systematic algorithm based on RL to search for a dynamics to attack and call this method env-searchbb. The algorithm is included in appendix. At each time step, an attacker proposes a change to the current environment dynamics with some perturbation ΔM, where ∥ΔM/M ∥ is bounded by some constant , and we find the new state ,M ′ at time step following the current policy under dynamics M ′ = M + ΔM, then the attacker agent will get reward˜= 1/∥ ,M ′ − ′ ∥. We demonstrate this in env-search-bb using DDPG [19] to train the attacker. In order to show that this method works better than random search, we also compare with the random dynamics search method, and keep the bound of maximum perturbation ∥ΔM/M ∥ the same.

Physical Attacks
We discuss how to apply previous proposed FD attack algorithms to generate adversarial perturbations on real world DRL models. We choose visual navigation robots as our experiment platform. There are several challenges to perform real-world attacks in this task. (1) Imperfect camera. Images observed by the robot usually are captured by cameras, and the computed perturbation may not be directly applicable on real world objects due to camera sensor noise and color shift. (2) Imperfect fabrication process. Using a printer to make the adversarial image patch limits the attacker to printable colors, and there exists a color gap between input image to the printer and output paper from the printer. (3) Imperfect alignment of the patch. Mounting the adversarial image patch at an exact position with a particular orientation is hard. Thus, the attack should be robust to variations of relative position/orientation of the image patch to the robot position. In order to overcome the above challenges, we adapt our algorithm to generate an adversarial patch that is robust against these imperfections. We select a discrete control task for visual navigation as our real world victim model (see details in experiment section).
In order to improve the robustness of the generated patch against various mounting positions, we randomly sample multiple binary masks { } 1 =1 and apply the masks to the state (an image) , such that any part of the patch can be used for attack. One mask is consisted of a rectangular region with value 1 and all other regions with value 0. Denote the generated perturbed image as ′ , then the masked frame masked with mask could be defined as: masked = ⊙ (1 − ) + ′ ⊙ , where ⊙ is the element-wise product.
To further improve robustness against imperfect printer and variability of mounting orientation, we randomly generate a set of transformations = { } 2 =1 including contrast, brightness adjustments, random rotation adjustments on masked and get the final image to be optimized: final = ( masked ). Define the function of generating the final image final as : Given a trained policy with parameters , the pristine optimal action output is * = ( ). We select a target action ′ , which should have smaller return than * . The objective function is shown as follows: ′ = arg min ′ = 1 , = 2

EXPERIMENTS
We design our experiments to answer the following questions: (1) Can our proposed black-box method achieve similar or better performance compared with existing white-box and black-box methods? (2) How does the adaptive SFD method perform compared with FD? (3) How much improvement of sample efficiency does online sequential attack obtain? (4) Does attacking the most important frame selection work better than attacking other frames? (5) Does the RL based environment dynamics attack perform better than random search? (6) Does the real robot attack work in the visual navigation task? To answer these questions, we first introduce the environments we use, and then introduce our baselines and evaluations on all methods.
Experiment environments and victim RL models. We attack several agents trained for five different simulated RL environments: Atari games including Pong and Enduro [5], HalfCheetah and Hopper in MuJoCo [36], and the driving simulation TORCS [29]. We train DQN [23] on Pong, Enduro and TORCS, and train DDPG [19] on HalfCheetah and Hopper. We report the cummulative reward on the first 500 frames. The reward function for TORCS comes from [29] and DQN network architecture comes from [23]. The network for continuous control using DDPG comes from [10].
Baselines. We compare the agents' performance under all attacks with their performance under no attack, denoted as non-adv. We test the white-box attacks with FGSM [14] (obs-fgsm-wb) and blackbox attacks with obs-fgsm-bb [16]  , where we use all of the first frames to compute the gradient for generating a perturbation and then apply the perturbation to the subsequent frames. We report the cumulative rewards of the first frames and the final cumulative reward among the first 500 frames by applying the perturbation after the first frames. We implement a baseline method obs-seq[F ]-rand-bb by using random noise as the perturbation to evaluate the effectiveness of our algorithms. To increase the attack efficiency, we also evaluate the obs-seq[L ]-fd-bb (L for "largest"), in which we select 20% of the first frames that have the largest value variance to generate the universal perturbation, and obs-seq[S ]-fd-bb (S for "smallest"), in which we select 20% of the first the frames that have the smallest value variance to generate the universal perturbation, as baseline.
Perturbations on components other than the observations. For attacks on environment dynamics, we test env-rand-bb and env-search-bb on MuJoCo and TORCS. In the tests on MuJoCo, we perturb the body mass and body inertia vector, which are in R 32 for HalfCheetah and R 20 for Hopper. In the tests on TORCS, we perturb the road friction coefficient and bump size, which is in R 10 . The perturbation strength is within 10% of the original magnitude of the dynamics being perturbed.
Real robot experiment. We conduct physical attack experiments on an Anki Vector robot [1]. For training the DRL policy, we design a discrete control task for the robot in a closed playground. The robot has a discrete action space of going forward, turning left, and turning right. It receives a positive reward of 3 for moving forward (in any direction) and a reward of −10 for colliding with anything. The task ends if the robot collides with the wall. We

Experimental Results
The first 6 columns of Table 2 shows the results of the attacks on TORCS, including baseline attacks: obs-fgsm-wb, and obs-fgsm-bb, and finite difference based methods (obs-fd-bb, obs-s[n]fd-bb) on black-box settings. It shows that obs-fd-bb could achieve similar performance compared with whitebox attack (obs-fgsm-wb) and slightly better than the baseline blackbox attack (obs-fgsm-bb). Moreover, for obs-s[ ]fd-bb, it shows the effectiveness and with increase, it will increase the computation cost but the attack effectiveness increases as well. For the results, we could observe that when = 40, it could achieve comparable attack effectiveness compared to obs-fd-bb, therefore, in the following experiments, we select = 40 when we report obs-sfd-bb. In Table 3, we show the number of queries for obs-s[n]fd-bb with different iteration parameter and the number of queries for obs-fd-bb. The results show that obs-sfd-bb uses significantly fewer queries (around 1000 to 6000) than obs-fd-bb (around 14,000) but achieves similar attack performance.  Table 2 with different ∞ norm bound ( = 0.05 and 0.1).
We could observe that the baseline obs-seq[F ]-rand-bb is not effective, while obs-seq[F ]-fd-bb achieves attack performance close to its white-box counterpart obs-seq[F ]-fgsm-wb and to nononline sequential attack obs-fd-bb. Even when = 10, the performance is still good. The 14-17 columns of Table 2 shows the results of optimal frame selection. We could observe that when we select a set of states with the largest Q value variance (obs-seq[L ]-fd-bb) to estimate the gradient, the attack is more effective than selecting states with the smallest Q value variance (obs-seq[S ]-fd-bb). It also empirically verifies corollary 1. When is very small ( = 10), the estimated universal perturbation may be not strong enough to apply to the following frames while when = 60, the attack performance is reasonably good and similar to obs-seq[F60]-fd-bb. Therefore, in the following settings, we select obs-seq[L60]-fd-bb as our default setting.
Finally, we combine online-sequential attack and sampling based finite difference together to evaluate the performance. We show the results of obs-seq[L60]-s[ ]fd-bb by selecting the 20% of frames with the largest Q value variance within the first 60 frames to estimate the gradient and using SFD with iterations. From Table 2, we could observe that it is clear that with more iterations; we are able to get more accurate estimation of the gradients and thus achieve better attack performance. When = 100, it could achieve comparable attack effectiveness. By looking at Table 3, we could find that when = 100, the number of queries for SFD is around 6k, which is still significantly smaller than needed for FD, which takes 14k queries to estimate the gradient on an image of size 84 × 84 (14112 = 84 × 84 × 2). Table 4: Results of different attacks on other environments. We report the cumulative reward within first 500 frames.  We provide the results of attack applied on observation space in other environments in Table 4. These environments include Atari games Pong and Enduro, and MuJoCo robotics simulation environments HalfCheetah and Hopper. The results further instantiates the effectiveness of the set of proposed FD methods.
Perturbations on components other than the observations. Attacks on environment dynamics. In Table 5, we show our results for performing targeted adversarial environment dynamics attack. The results are the 2 distance to the target state (the smaller the better). The results show that random search method performs worse than RL based search method in terms of reaching a specific state after certain steps. The quality of the attack can be qualitatively evaluated by observing the sequence of states when the agent is being attacked and see whether the target state has been achieved. The results are shown in figure 3. We could observe that the final stages of our RL based method (env-search-bb) are similar to the targeted state among different games. We include example trajectories of agents under dynamics attack in our appendix.    Real robot policy attack To evaluate the performance of the adversarial patch in the physical world, we print it out and mount it at five different positions (front, left, right, up and below of the robot position) and put the robot at 50 different viewing angles from the adversarial patch (varied from -60 • to 60 • ). We select a target action for attack, which results in suboptimal behavior such as a collision. The physical experiment settings are included in Figure 2. The results of attack success rate are shown in Table 6. The results show that by mounting the patch just in front of the robot results in the best attack success rate both in white-box attack and black-box attack. When there are inaccuracies of patch mounting, such as mounting the patch to the slight left, right, up or below of the robot field of view, the attack success rate goes down.

DISCUSSION AND CONCLUSIONS
We propose and evaluate a set of black-box adversarial attacks on DRL models using finite-difference methods. The sample efficiency of FD based methods are further improved by using the adaptive sampling method, online sequential attack and attack frame selection. Most importantly, we provide the first example of adversarial attacks on real world visual navigation robot. Studying the adversarial attack on real robot can help to find potential defense approaches to improve the robustness of DRL models. From the perspective of training robust RL policies, it is important to know the severeness of the risk related with the proposed attacks. Among the proposed attacks, the environment dynamics attack can be a more realistic potential risk to consider than the attacks based on observations. The reason is that this attack does not require access to modify the policy network software system and only requires access to modify environment dynamics, and by modifying the environment dynamics parameters such as changing road condition in autonomous driving, we see from our experiments that the agent tends to fail with the original policy. The observation space based attack, especially the black-box attacks, are also important to defend against, since an attacker can definitely query the network and may have access to change the observations. The real world physical attack experiment by modifying the environment poses a real concern for DRL policies that may be potentially affected. We hope our exploratory work and the analysis of attacks help form a more complete view for what threats should be considered in ongoing research in robust reinforcement learning.

A ADAPTIVE SAMPLING-BASED FD ALGORITHM
We give the SFD algorithm description in the following Algorithm 1. : well trained policy : the environment used to train the original policy : total training steps : target attack state : maximum episodic steps Initialization: : RL policies for generating dynamics perturbations : replay buffer = .dynamics For = 0 to = .sample_action() .change_dynamics( ) reached_state = evaluate_policy( , ) reward = 1/∥reached_state-∥ done = episode steps reach action = B.append( , , action, reward, done) train_policy( , B) = Details of dynamics attack on Torcs environment. In the Torcs self-driving car environment, we modified the road friction parameter and road bump size to generate novel challenging environments. Specifically, parameters within the XML files that defines these road properties will be modified and recompiled each time a new environment is generated.
C PROOF OF LEMMA 1 Proof. Following the notation in Lemma 1, define Note that the randomness is with respect to the dimensions. These probabilities are, when we randomly choose dimensions in a gradient vector of dimensions, the chance of sampling some dimension with gradient absolute value no less than + or , respectively. For ∇ ( ), its neighbor gradient's absolute value follows the following distribution: Define 0 = random ( + ) , 0 = [ random ( )− random ( + )] as the number of nontrivial gradients estimated in iteration 0, and define as the number of nontrivial gradients estimated in iteration , then the ratio = + −1 + −1 characterizes in every iteration, the ratio of nontrivial gradient estimation over the total number of gradient estimation for ≥ 1, while random ( ) characterizes nontrivial gradient estimation ratio if using random sampling method. For = 0, 0 = random ( ). Since is the ratio of the number non-trivial gradients over the total number of gradients estimated in the -th iteration, if we randomly select the dimensions, will be the same as random ( ). Using our SFD method, if for every > 0, we have ≥ random ( ), then overall speaking, the ratio of the number of non-trivial gradients estimated over the total number of gradients estimated, SFD ( ), will be greater than random ( ). The definition of SFD ( ) is where is the number of iterations. We now prove that if 1 ≤ 1 − random ( ), then for > 0, More specifically, for = 0, since we perform uniform random sampling, and we sample dimensions, we have Then if we can prove that for ≥ 1, ≥ random ( ), and for some , we have > random ( ), it in turn proves SFD ( ) > random ( ). The reason is that if for every step in our SFD algorithm the sample efficiency is at least the same as random sampling, and for some iterations our SFD is more efficient, then overall speaking our SFD based gradient estimation is more efficient than random sampling based gradient estimation. Now from the above definition, we have Therefore, when 1 < 1 − random ( ), > random ( ), our sampling algorithm is more efficient than random sampling. □

D PROOF OF THEOREM 2
Proof. when ∈ R 1 , assume function is ∞ , by Taylor's series we have Combine the two equations we get which means the truncation error is bounded by ( 2 ). Moreover, we have where = sup ∈ [ − 0 , + 0 ] (3) ( ) 6 , and 0 < ≤ 0 . We can regard each dimension as a single variable function ( ), then we have Then ∀ ∈ {1, 2, · · · , }, assume we are able to sample all nontrivial gradients with absolute gradient value no less than , then we have ∑︁ Therefore, the truncation error of gradients estimation is upper bounded by the following inequality.
for some > 0, and ∇ˆ( ) is the estimated gradient of with respect to . □

E PROOF OF COROLLARY 3
Proof. Recall the definition of Q value is The variance of Q value at a state is defined as where A is the action space of the MDP, and |A| denotes the number of actions. Suppose we are to attack state and state where the Q value variance of this two states are ( ( ), ·) and ( ( ), ·), and assume < . Denote the state-action pair Q values after attack are ( ,ˆ) and ( ,ˆ), respectively. During the attack, state is modified toˆ, and state is modified toˆ, and their action's Q-value also change, so we useˆandˆto denote the actions after the attack. By using and instead ofˆandˆ, we mean that though the observed states are modified by the attack algorithm, but the true states do not change. By using a different action notation, we mean that since the observed states have been modified, the optimal actions at the modified states can be different from the optimal actions at the original observed states. Then the total discounted expected return for the entire episode can be expressed as (assume all actions are optimal actions) Since < , ′′ can also be expressed as Subtract ′ by ′′ we get According to our claim that states where the variance of Q value function is small will get better attack effect, suppose ( ( )) > ( ( )), and assume the range of Q value at step is larger than step , then we have Therefore ′ − ′′ < 0 which means attack state the agent will get less return in expectation. If ( ( )) < ( ( )), assume the range of Q value at step is smaller than step , then we have If − is very small or  1). For both models, we added ReLU activation layers between these fully connected layers.
The TORCS autonomous driving environment is a discrete action space control environment with 9 actions, they are turn left, turn right, keep going, turn left and accelerate, turn right and accelerate, accelerate, turn left and decelerate, turn right and decelerate and decelerate. The other 4 games, Pong, Enduro, HalfCheetah, and Hopper are standard OpenAI gym environment.
The trained model's performance when tested without any attack is included in the following Table 7. The DDPG neural network used for env-search-bb is the same as the first model (3-layer fully connected network) used for training the policy for HalfCheetah, except that the input dimension is of the perturbation parameters' dimension, and output dimension is also of the perturbation parameters' dimension. For HalfCheetah, Hopper and TORCS, these input and output dimensions are 32, 20, and 10, respectively.

G ADDITIONAL EXPERIMENT RESULTS
Example Trajectory of Agents Under Dynamics Attack. In Figure 4, Figure 5, we show the sequences of states when the agents are under attack with the random search or reinforcement learning based search method. The last image in each sequence denotes the state at same step . The last image in each abnormal dynamics rollout sequence corresponds to the target state, the last image in the attacked dynamics using RL search denotes the attacked results using env-search-bb, and the last image in the attacked dynamics using random search denotes the attacked results using env-randbb. It can be seen from these figures that env-search-bb method is very effective at achieving targeted attack while using random search, it is relatively harder to achieve this.
Real world attack additional results. We include in Figure 6 the original robot observation, the optimized perturbed image, the actual printed perturbed image patch and the bird-view of the robot environment after the patch has been mounted. , optimized perturbed image with white-box approach (second), optimized perturbed image with black-box approach (third), printed perturbation patch with white-box approach (forth), printed perturbation patch with black-box approach (fifth) physical environment with the playground, adversarial patch and the robot (sixth).