{'fraction_non_alphanumeric': 0.03335386273528619, 'fraction_numerical': 0.009680896108395242, 'mean_word_length': 4.456158773495747, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications

One major factor impeding more widespread adoption of deep neural networks (DNNs) is their issues with robustness, which is essential for safety critical applications such as autonomous driving. This has motivated much recent work on adversarial attacks for DNNs, which mostly focus on pixel-level perturbations void of semantic meaning. In contrast, we present a general framework for adversarial black-box attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks. To do this, we re-frame the adversarial attack problem as learning a distribution of parameters that always fool the agent. In the semantic case, our proposed adversary (denoted as BBGAN) is trained to sample parameters that describe the environment with which the black-box agent interacts, such that the agent performs its dedicated task poorly in this environment. We apply BBGAN on three different tasks (primarily targeting aspects of autonomous navigation): object detection, self-driving, and autonomous UAV racing. On these tasks, BBGAN can generate failure cases that consistently fool an agent. We also demonstrate the usefulness of our framework as an analysis tool by visualizing systemic failure cases and uncovering semantic insights about the agents themselves.


Introduction
As a result of recent advances in machine learning and computer vision, deep neural networks (DNNS) are now interleaved with many aspects of our daily lives. DNNs are used to suggest news articles to read and movies to watch, automatically edit our photos and videos, and translate between hundreds of languages. They are also bound to disrupt transportation with autonomous driving slowly becoming a reality. While there are already impressive demos and some successful deployments, safety concerns for boundary conditions persist. While current models work very well on average, they struggle with robustness in certain cases. Recent work in the adversarial attack literature shows how weather geometry camera Figure 1: Semantic Adversarial Diagnostic Attacks. Neural networks can perform very well on average for a host of tasks; however, they do perform poorly or downright fail when encountering some environments. To diagnose why they fail and how they can be improved, we seek to learn the underlying distribution of semantic parameters, which generate environments that pose difficulty to these networks when applied to three safety critical tasks: object detection, self-driving cars, and autonomous UAV racing.
sensitive DNNs are to input noise. These attacks usually utilize the information about the network structure to perform gradient updates in order to derive targeted perturbations (coined white-box attacks). These perturbations are injected into the input image at the pixel-level, so as to either confuse the network or enforce a specific behavior [45,18,9,24]. There are also attacks that assume the DNN to be a blackbox (i.e. undisclosed network architecture), but they still operate at the pixel-level [11,33]. Targeted pixel attacks can lead to severe performance degradation not only in classification, but also on tasks like object detection [12], face recognition [43], and segmentation [50].
In practice, such pixel attacks are much less likely to naturally occur than semantic attacks which include changes in camera viewpoint, lighting conditions, street layouts, etc. The literature on semantic attacks is much sparser, since they are much more subtle and difficult to analyze [53]. Yet, this type of attack is critical to understand/diagnose failure cases that might occur in the real-world. While it is very difficult to investigate semantic attacks on real data, we can leverage simulation as a proxy that can unearth use-ful insights transferable to the real-world. Figure 1 shows an example of an object misclassified by the YOLOV3 detector [38] applied to an image generated by Blender [7], an autonomous UAV racing [31] failure case in a recently developed general purpose simulator (Sim4CV [30]), and an autonomous driving failure case in a popular driving simulator (CARLA [14]). These failures arise from adversarial attacks on the semantic parameters of the environment.
In this work, we consider environments that are adequately photo-realistic and parameterized by a compact set of variables that have direct semantic meaning (e.g. camera viewpoint, lighting/weather conditions, road layout, etc.). Since the generation process of these environments from their parameters is quite complicated and in general nondifferentiable, we treat it as a black-box function that can be queried but not back-propagated through. We seek to learn an adversary that can produce fooling parameters to construct an environment, in which the agent (which is also a black-box) fails in its task. Unlike most adversarial attacks that generate sparse instances of failure, our proposed adversary provides a more comprehensive view on how an agent can fail; we learn the distribution of fooling parameters for a particular agent and task and sample from it. Since Generative Adversarial Networks (GANs [17,4]) have emerged as a promising family of unsupervised learning techniques that can model high-dimensional distributions [52,21,20,25], we model our adversary as a GAN, appropriately denoted as black-box GAN (BBGAN).

Contributions.
(1) We formalize adversarial attacks in a more general setup to include both semantic and conventional pixel attacks. (2) We propose BBGAN in order to learn the underlying distribution of semantic adversarial attacks. (3) We show promising results on three different safety-critical applications used in autonomous navigation, thus, illustrating the usefulness of our approach in analyzing agent failure, along with some analysis and insights about transferability of these attacks.

Related Work
Pixel Attacks on Classifiers. The work of Szegedy [45] was the first to introduce a formulation of attacking neural networks as an optimization. The method produces a minimal perturbation of the image pixels that fools a trained classifier (incorrect predictions). Several works followed the same approach but with different formulations, such as Fast Gradient Sign Method (FGSM) [18] and Projected Gradient Descent [24,28]. The notable work of [9] gives a comprehensive study of different ways to fool the network with minimal pixel perturbation. However, all these methods are limited to pixel perturbations and only fool classifiers, while we consider more general cases of attacks, e.g. changes in camera viewpoint to fool a detector or change in parametric environment with which an agent A interacts. The agent receives an observation ot from the environment and produces an action at. The environment scores the agent and updates its state until the episode finishes. A final score Q(A, Eµ) is given to the adversary G, which in turn updates itself to propose more adversarial parameters µ for the next iteration. weather conditions to fool a self-driving car.
Most of these attacks are white-box attacks, in which the algorithm has access to network gradients. However, some methods generate black-box attacks, which only have access to the classifier output. Most of these methods approximate the gradient to employ techniques similar to white-box attacks, e.g. through finite differences [11] or PCA and randomized dimension selection. In this work, our core attack is black-box in nature, but we are interested in the distribution of the semantic parameters that fool the agent, more so than individual fooling examples. This limits the usefulness of approximating the gradient to reach a single solution.
Attacks beyond Classifiers and Pixels. Image classification is not the only application for adversarial pixel attacks. Other tasks include object detection [39] and semantic segmentation [50], where the image pixels are perturbed through additive components or spatial transformations. Beyond pixel perturbations, Alcorn et al. attack famous classifiers and detectors by changing object pose [3], while Zeng et al. [53] attack deep classifiers and VQA (Visual Question and Answering) agents by perturbing lighting and surface normals. They show that most common image space adversarial attacks are not natural and cannot be realized in real 3D scenes. Inspired by this insight, we tackle the topic of semantic attacks by using readily available virtual environments with plausible 3D setups to systematically test the agents. In fact, our formulation includes attacks not only on static agents like object detectors, but also agents that interact with dynamic environments, such as self-driving agents. To the best of our knowledge, this is the first work to introduce adversarial attacks in CARLA [14], a standard autonomous navigation benchmark.
Adversarial Attacks and Reinforcement Learning. We formulate a black-box adversarial attack in a general setup that includes an environment that rewards an agent for some task. An adversary outside the environment is tasked to fool the agent by modifying the environment. Our formulation is naturally inspired by Reinforcement Learning (RL), in which agents can have multiple actions and receive partial rewards as they proceed in their task [44]. However, in RL, the agent is the focus of learning and the goal is to achieve a task in the environment. In adversarial attacks, the agent is usually fixed and learning is performed by the adversary in order to fool the agent or diagnose it. Recent work in [51] introduces an RL environment update as a way to improve the performance of agents inside this environment. Other works suggest the opposite, i.e. the environment is updated to fool the agent for training robust RL agents [29,36]. In contrast, our virtual environment is parameterized by semantic attributes with the goal of fooling and diagnosing the failures of an already trained agent.

Methodology
Typical adversarial pixel attacks involve a neural network agent C (e.g. classifier or detector) that takes an image x ∈ [0, 1] n as input and outputs a multinoulli distribution over K class labels with softmax values [l 1 , l 2 , ..., l K ], where l j is the softmax value for class j. The adversary (attacker) tries to produce a perturbed image x ∈ [0, 1] n that is as close as possible to x, such that C changes its class prediction from x to x . The objective to be optimized is:

Generalizing Adversarial Attacks
Extending attacks to general agents. In this work, we generalize the adversarial attack setup beyond pixel perturbations. Our more general setup (refer to Figure 2) includes semantic attacks, e.g. perturbing the camera pose or lighting conditions of the environment that generates observations (e.g. pixels in 2D images). An environment E µ is parametrized by µ ∈ [µ min , µ max ] d . It has an internal state s t and produce observations o t ∈ R n at each time step t ∈ {1, . . . , T }. The environment interacts with a trained agent A, which takes o t from E µ to produce actions a t . At each time step t and after the agent performs a t , the internal state of the environment is updated: s t+1 = E µ (s t , a t ). The environment rewards the agent with r t = R(s t , a t ), for some reward function R. We define the episode score Q(A, E µ ) = T t=1 r t of all intermediate rewards. The goal of A is to complete a task by maximizing Q. Outside the black-box (environment and agent), the adversary G attacks A by modifying E µ through its parameters µ.
Distribution of Adversarial Attacks. We define P µ to be the fooling distribution of semantic parameters µ representing environment E µ , which fools the agent A.
Here, is a small threshold that is defined for each adversarial attack. Distribution P µ ensures both exploration and exploitation, since it covers all failure cases of A and still results in successful attacks for all its samples. Its PDF is unstructured and depends on the complexity of the agent. We seek an adversary G that tries to learn P µ , so it can be used to comprehensively analyze the weaknesses of A. Unlike the common practice of finding individual adversarial examples (most often images), we address the attacks distribution-wise in a compact semantic parameter space. We denote our analysis technique as Semantic Adversarial Diagnostic Attack (SADA). It is semantic because of the nature of the environment parameters and diagnostic because a fooling distribution is sought. Section 5.4 highlights what these semantic attacks can reveal about the nature of the agents used in safety-critical tasks (e.g. object detectors and autonomous navigation agents).
For the adversary G to achieve this challenging goal, we propose to optimize the following objective: Algorithm 1 describes a general setup for G to learn to generate fooling parameters. It also includes a mechanism for evaluating G in the black-box environment E µ for L iterations after training it to attack agent A. An attack is considered a fooling attack, if parameter µ sampled from G achieves an episode score Q(A, E µ ) ≤ . Consequently, the Attack Fooling Rate (AFR) is defined as the rate at which samples from G are fooling attacks. In addition to AFR, the algorithm returns the set S µ of adversarial examples that can be used to diagnose the agent. The equality constraint in Eq (3) is very strict to include all the fooling parameters µ of the fooling distribution. It acts as a perceptuality metric in our novel generalized attack to prevent unrealistic attacks. In the following, we relax this equality constraint and leverage recent advances in GANs to learn an estimate of the distribution P µ .

Black Box Generative Adversarial Network
Generative Adversarial Networks (GANs) are a promising family of unsupervised techniques that can model complex domains, e.g. natural images [17,4,19]. GANs consist of a Discriminator D x and a Generator G x that are adversarially trained to optimize the loss L GAN (G x , D x , P X ) in Eq (4), where P X is the distribution of images in domain X and z ∈ R c is a latent random Gaussian vector. We learn an adversary G, which samples semantic parameters µ that parametrize the environment Eµ, such that an agent A fails in a given task in Eµ. The inducer produces the induced set S µ from a uniformly sampled set Ω by filtering the lowest scoring µ (according to Q value), and passing S µ for BBGAN training. Note that Q1 ≤ Qs..., ≤ QN , where s = S µ , N = |Ω|. The inducer and the discriminator are only used during training (dashed lines), after which the adversary learns the fooling distribution P µ . Three safety-critical applications are used to demonstrate this in three virtual environments: object detection (in Blender [7]), self-driving cars (in CARLA [14]), and autonomous racing UAVs (in Sim4CV [30] D x tries to determine if a given sample (e.g. image x) is real (from the training dataset) or fake (generated by G x ). On the other hand, G x tries to randomly generate samples that fool D x (e.g. misclassification). Both networks are proven to converge when G x can reliably produce the underlying distribution of the real samples [17].
In this paper, we propose to learn the fooling distribution P µ using a GAN setup, which we denote as black-box GAN (BBGAN). We follow a similar GAN objective but replace the image domain x by the semantic environment parameter µ. However, since we do not have direct access to P µ , we propose a module called the inducer, which is tasked to produce a parameter set S µ (induced set) that belongs to P µ . Thus, this setup relaxes Eq (3) to: So, the final BBGAN loss becomes: Here, G µ is the generator acting as the adversary, and z ∈ R m is a random variable sampled from a normal distribution. A simple inducer can be just a filter that takes a uniformly sampled set Ω = {µ i ∼ Uni([µ min , µ max ])} i=N i=1 and suggests the lowest Q-scoring µ i that satisfies the condition Q(µ i ) ≤ . The selected samples constitute the induced set S µ . The BBGAN treats the induced set as a training set, dataset Φ containing all images and their true class label simulation environment partially described by µ that A navigates in for a target Observation ot attacked image after added noise  so the samples in S µ act as virtual samples from the fooling distribution P µ that we want to learn. As the induced set size S µ increases, the BBGAN learns more of P µ . As |S µ | → ∞, any sample from S µ becomes a sample of P µ and the BBGAN in Eq (6) satisfies the strict problem in Eq (3). Consequently, sampling from G µ would consistently fool agent A. We show an empirical proof for this in the appendix and in Section 5 we show how we consistently fool three different agents by G µ samples. The number of samples needed for S µ to be representative of P µ depends on the dimensionality d of µ. Because of the black-box and stochastic nature of E µ and A (similar to other RL environments), we follow the random sampling scheme common in RL [26] instead of deterministic gradient estimation. In Section 5.3, we compare our method against baselines that use different approaches to solve Eq (3).

Special Cases of Adversarial Attacks
One can show that the generic adversarial attack framework detailed above includes well-known types of attacks as special cases, summarized in Table 1. In fact, the general setup allows for static agents (e.g. classifiers and detectors) as well as dynamic agents (e.g. an autonomous agent acting in a dynamic environment). It also covers pixel-wise image perturbations, as well as, semantic attacks that try to fool the agent in a more realistic scenario. The generic attack also allows for a more flexible way to define the attack success based on an application-specific threshold and the agent score Q. In the appendix, we provide more details on the inclusiveness of our generic setup.

Object Detection
Object detection is one of the core perception tasks commonly used in autonomous navigation. In short, its goal is to determine the bounding box and class label of objects in an image. You Only Look Once (YOLO) object detectors popularized a single-stage approach, in which the detector observes the entire image and regresses the boundaries of the bounding boxes and the classes directly [37]. This trades off the accuracy of the detector for speed, making real-time object detection possible (passing the 30-FPS threshold).
Agent. Based on its suitability for autonomous applications, we choose the very fast, state-of-the-art YOLOv3 object detector as the agent in our SADA framework [38].
Environment. We use Blender [7] to construct a scene based on freely available 3D scenes and CAD models from free3D [2] and 3Dwarehouse [1]. The scene was picked to be an urban scene with an open area to allow for different rendering setups. The scene includes one object of interest, one camera, and one main light source all directed toward the center of the object. The light is a fixed strength spotlight located at a fixed distance from the object. The material of each object is semi-metallic, which is common for the classes under consideration. The 12 object classes (10 from Pascal-3D [49] and 2 from ShapeNet [10]) are: aeroplane, bench, bicycle, boat, bottle, bus, car, chair, dining table, motorbike, train, and truck. The total number of shapes across all 12 classes is 100 as shown in Figure 2. We augment Pascal3D shapes with others from Shapenet [10] and Modelnet40 [48] from the same class. At each iteration, one shape from the intended class is randomly picked and placed in the middle of the scene. Then, the Blender rendered image is passed to YOLOV3 for detection. Please refer to the appendix for more details on the dataset.
Environment parameters. We take µ ∈ R 8 , defining parameters that have shown to affect detection performance and frequently occur in real setups (refer to Figure 4). The object is centered in the virtual scene, and the camera circles around the object keeping the object in the center of the rendered image.

Self-Driving
There is a lot of recent work in autonomous driving especially in the fields of robotics and computer vision [8,15,34,13]. In general, complete driving systems are very complex and difficult to analyze or simulate. By learning the underlying distribution of failure cases, our work provides a safe way to analyze the robustness of such a complete system. While our analysis is done in simulation only, we would like to highlight that sim-to-real transfer is a very active research field nowadays [35,40,41,42,46]. Agent. We use an autonomous driving agent (based on CIL [13]), which was trained on the environment E µ with default parameters. The driving-policy was trained end-to-end to predict car controls given an input image and is conditioned on high-level commands (e.g. turn right at the next intersection) in order to facilitate autonomous navigation.
Environment. We use the recent CARLA driving simulator [14], the most realistic open-source urban driving simulator currently available. We consider the three common tasks of driving in a straight line, completing one turn, and navigating between two random points [14]. The score is measured as the average success of five pairs of start and end positions.
Environment parameters. Since experiments are timeconsuming, we restrict ourselves to three parameters, two of which pertain to the mounted camera viewpoint and the third controls the appearance of the environment by changing the weather setting (e.g.'clear noon', 'clear sunset', 'cloudy after rain', etc.). As such, we construct an environ-ment by randomly perturbing the position and rotation of the default camera along the z-axis and around the pitch axis respectively, and by picking one of the weather conditions. Intuitively, this helps measure the robustness of the driving policy to the camera position (e.g. deploying the same policy in a different vehicle) and to environmental conditions.

UAV Racing
In recent years, UAV (unmanned aerial vehicle) racing has emerged as a new sport where pilots compete in navigating small UAVs through race courses at high speeds. Since this is a very interesting research problem, it has also been picked up by the robotics and vision communities [23,32]. Agent. We use a fixed agent to autonomously fly through each course and measure its success as percentage of gates passed [32]. If the next gate was not reached within 10 seconds, we reset the agent at the last gate. We also record the time needed to complete the course. The agent uses a perception network that produces waypoints from image input and a PID controller to produce low-level controls. Environment. We use the general-purpose simulator for computer vision applications, Sim4CV [30]. Sim4CV is not only versatile but also photo-realistic and provides a suitable environment for UAV racing. Environment parameters. Here, we change the geometry of the race course environment. We define three different race track templates with 3-5 2D anchor points, respectively. These points describe a second order B-spline and are perturbed to generate various race tracks delineated by colored cones and populated by uniformly spaced gates. Please refer to the appendix for more details and visualizations of the generated tracks.

BBGAN
In this section, we show how our proposed BBGAN can be used to estimate the distribution of failure cases with respect to some environment parameters µ that affect the agent observations of the environment. Training. To learn the fooling distribution P µ , we train the BBGAN as described in Section 3.2. For this, we use a vanilla GAN model [17], but any GAN architecture can be used. We use a simple FCN with 2 layers for the Generator G and Discriminator D. We train the GAN following convention, but since we do not have access to the true distribution that we want to learn (i.e. real samples), we induce the set by randomly sampling N parameter vector samples µ, and then picking the K worst among them (according to Q score). For object detection, we use N = 20000 image renderings for each class (a total of 240K images), as described in Section 4.1. Due to the computational cost, our  Boosting. Inspired by the classical Adaboost algorithm [16], we use a boosting strategy to improve the performance of our BBGAN. Our boosting strategy simply utilizes the samples generated by the previous stage adversary G k−1 in inducing the training set for the current stage adversary G k . This is done by including the generated samples to Ω before training G k . This strategy can be applied iteratively in a sequential manner more than once. The intuition here is that the main computational burden in training the BBGAN is not the GAN training itself, but computing the agent episodes, each of which can take multiple hours for the case of self-driving. For more details, including the algorithm, a mathematical justification and more experimental results please refer to the the appendix.

Testing, Evaluation, and Baselines
To highlight the merits of BBGAN, we seek to compare it against baseline methods (see below), which also aim to estimate the fooling distribution P µ . In this comparative study, each method produces M fooling/adversarial samples (250 for object detection and 100 for self-driving and UAV racing) based on its estimate of P µ . Then, the attack fooling rate (AFR) for each method is computed as the percentage of the M adversarial samples that fooled the agent. To determine whether the agent is fooled, we use a fooling rate threshold = 0.3 [12], = 0.6, and = 0.7 for object detection, self-driving, and UAV racing, respectively. In the following, we briefly explain the baselines.
Random. We uniformly sample random parameters µ within an admissible range that is application dependent.
Gaussian Mixture Model (GMM). We fit a full covariance GMM of varying Gaussian components to estimate the distribution of the samples in the induced set S µ . The variants are denoted as Gaussian (one component), GMM10% and GMM50% (number of components as percentage of the samples in the induced set).
Bayesian Expected Improvement. We use the hyperopt [6] implementation of the Expected Improvement (EI) Bayesian Optimization algorithm [22] (similarly adopted by [47]) to minimize the score Q for the agent. The optimizer runs for 10 4 steps and it tends to gradually sample more around the global minimum of the function. So, we use the last N = 1000 samples to generate the induced set S µ and then learn a GMM on that with different Gaussian components. Finally, we sample M parameter vectors from the GMMs and report results for the best model. The advantage of this baseline over GMM is that the samples are more structured and usually clustered around the global minimum, but this happens after a long expensive trace which prevents using it for the autonomous applications.
Multi-Class SVM. We bin the output score Q into 5 equally sized bins and train a multi-class SVM classifier on the complete set Ω to predict the correct bin. We then randomly sample parameter vectors µ, classify them, and sort them by the predicted score. We pick the M samples with the lowest Q score.
Gaussian Process Regression. Similar to the SVM case, we train a Gaussian Process Regressor [27] with an exponential kernel to regress Q scores from the corresponding µ parameters that generated the environment on the dataset Ω. Table 2 summarizes the AFR results for the aforementioned baselines and our BBGAN approach across all three applications. For object detection, we show 3 out of 12 classes and report the average across all classes. For autonomous driving, we report the results on all three driving tasks. For UAV racing, we report the results for three different track types, parameterized by an increasing number of 2D anchor points (3, 4 and 5) representing µ (refer to Section 4.2 and Section 4.3).

Results
Our results show that we consistently outperform the baselines, even the ones that were trained on the complete Ω rather than the smaller induced set S µ , such as the multiclass SVM and the GP regressor. While some baselines perform well on the autonomous driving application where µ consists of only 3 parameters, our approach outperforms them by a large margin on the tasks with higher dimensional µ (e.g. object detection and UAV 5-anchors). To ensure that mode collapse (a GAN phenomenon where the generator collapses to generate a single point because of imbalance loss with the discriminator) does not occur, we do the following. (1) We visualize the individual µ distributions of the generated samples (as in Figure 5) and ensure the variety of the samples. (2) We measure the average standard deviation per parameter dimension to make sure it is not zero for BBGAN (would be less than some of other baselines). (3) We visualize the images/tracks created by these parameters as in Figure 6.

Analysis
The usefulness of SADA lies in that it is not only an attacking scheme using BBGAN, but it can also serve as a diagnosis tool to assess the systematic failures of agents. Diagnosis. To identify weaknesses and cases that result in systematic failure for the YOLOv3 detector, we fix some semantic parameters (distance to camera and object color) and attack the others (the four angles in Figure 4). We focus on two object classes that are relevant to autonomous driving: cars and motorbikes. We use rough models with homogeneous texturing (obtained from the training in Section 4.1) as well as detailed textured models. Figure 5 visualizes the learned fooling distribution for a detailed car model. Transferability. To demonstrate the transferability of the fooling parameter distribution to the real-world, we photograph a toy car using a standard mobile phone camera and use office desk lights to simulate the light source of the virtual environment. We orient the camera and the lighting source according to the fooling distribution learned from the virtual world. In Figure 5, we present a fooled real-world  photo of the toy car and the rendered virtual-world image, both of which are similarly fooled with the same erroneous class label. Please refer to the appendix for a similar analysis of the motorbike class, the transfer of attacks between different CAD models and the effect of occlusion (as yet another environment parameter) on detection.

Insights and Future Work
Object Detection with YOLOV3. In this case, we consistently found that for most objects, top-rear or top-front views of the object tend to fool the YOLOV3 detector. The color of the object does not play a significant role in fooling the detector, but usually colors that are closer to the background color tend to be preferred by the BBGAN samples (as shown in the qualitative examples).
Self-Driving. In this case, we found that weather is the least important parameter for fooling the driving policy, which indicates that the policy was trained to be insensitive to this factor. Interestingly, we observe that the learned policy is very sensitive to slight perturbations in the camera pose (height and pitch), indicating a systemic weakness that should be ratified with more robust training.
UAV Autonomous Navigation. We observe that the UAV fails if the track has very sharp turns. This makes intuitive sense and the results that were produced by our BBGAN consistently produce such tracks. While the tracks that are only parameterized by three control points can not achieve sharp turns, our BBGAN is still able to make the UAV agent fail by placing the racing gates very close to each other, thereby increasing the probability of hitting them.
Future Work. In the future, we plan to investigate defense mechanisms against semantic attacks described in this work. Our method can be used to identify difficult samples that lie in the fooling distribution. These samples can then be used to adversarially train more robust agents (e.g. classifiers, detectors, and autonomous navigation agents).

A. Empirical Justification for BBGAN
We want to show that as the size of the induced set |S µ | → ∞, learning an adversary according to the BB-GAN objective in Eq (7) converges to the fooling distribution of semantic parameters P µ defined in Eq (8).

A.1. Empirical Proof
We use the definition of Exhaustive Search (Algorithm 3.1) from the Audet and Hare book on derivative-free and black box optimization [5]. In this algorithm, we try to optimize an objective f : R d → R defined on a closed continuous global set Ω by densely sampling a countable subset S = {µ 1 , µ 2 , ..., µ N } ⊂ Ω. Theorem 3.1 [5] states that as long as the exhaustive search continues infinitely from the set S , the global solutions of f can be reached. Let's assume the global solutions µ * of f exists and defined as follows.
Let's denote the best solution reached up to the sample µ N to be µ best N . If the set S µ * is the set of all global solutions µ * , then Theorem 3.1 [5] states that Now let f (µ) = max(0, Q(E µ , A) − ) and let Ω = [µ min , µ max ], then the global solutions of the optimization: satisfy the two conditions in Eq (8) as follows.
Hence, the set of all global solutions includes all the points in the fooling distribution: Therefore, as the sampling set size |S | → ∞, all the points µ that lead to Q(E µ , A) ≤ , achieve the minimum objective in Eq (11) of zero and the set of best observed values |{µ best N }| → ∞. This set is what we refer to as the induced set S µ . From Eq (10) and Eq (13), we can infer that the induced set will include all fooling parameters as follows.
Finally if the set S µ has converged to the distribution P µ and we use S µ to train the BBGAN in Eq (7), then according to proposition 2 from the original GAN paper by Goodfellow et al. [17], the adversary G µ has learnt the distribution P µ and hence satisfies the following equation: This concludes our empirical proof for our BBGAN.

B. Special Cases of Generic Adversarial attacks:
In Table 3, we summarize the substitutions in the generic adversarial attack to get different special cases of adversarial attacks. In summary, the generic adversarial attack allows for static agents (like classifiers and detectors) and dynamic agents (like an autonomous agent acting in a dynamic environment). It also covers the direct attack case of pixel perturbation on images to attack classifiers, as well as semantic attacks that try to fool the agent in a more realistic scenario (e.g. camera direction to fool a detector). The generic attack also allows for a more flexible way to define the attack success based on an application-specific threshold and the agent score Q. In the following we provide detailed explanation and mathematical meaning of the substitutions.

B.1. Pixel Adversarial Attack on Image Classifiers
Probably the most popular adversarial attack in the literature is a pixel-level perturbation to fool an image classifier. This attack can be thought of as a special case of our general formulation. In this case, the agent A is a classifier C : [0, 1] n → [l 1 , l 2 , ..., l K ] and the environment E µ is a dataset containing the set Φ of all images in the classification dataset along with their respective ground truth labels, and y i ∈ {1, . . . , K}. The softmax value of the true class is given by l yi = max C(x i ). Parameter µ defines the fooling noise to be added to the images (i.e. d = n). The observation is simply an image from Φ with noise added: o t = x i + µ for some i ∈ {1, 2, .., |Φ|}. In classification, the environment is static with T = 1. To ensure the resulting image is in the admissible range, the noise added µ should fall in the range [−x i,min , 1 − x i,max ], where x i,min , x i,max are the min and max pixel value for the image x i . The sole action a 1 is simply the softmax score of the highest scoring class label predicted by C that is not the true class y i . Formally, a 1 = max j =yi C(x i + µ) = l j . The reward function is R(s 1 , a 1 ) = Q(C, Φ) = max(l yi − l j , 0). Here, = 0 for the classifier fooling to occur , which means fooling occurs if l yi − l j ≤ 0. Using these substitutions in the hard constraint in Eq (8) translates to the following constraints on the perturbed image.
For a single image attack, we can rewrite Eq (16) as follows: We observe that the constraints in Eq (17) become the following constraints of the original adversarial pixel attack formulation on a classifier C.

B.2. Semantic Adversarial Attack on Object Detectors
Extending adversarial attacks from classifiers to object detectors is straight-forward. We follow previous work [12] in defining the object detector as a function F : [0, 1] n → (R N ×K , R N ×4 ), which takes an n-dimensional image as input and outputs N detected objects. Each detected object has a probability distribution over K class labels and a 4-dimensional bounding box for the detected object. We take the top J proposals according to their confidence and discard the others. Analyzing the detector in our general setup is similar to the classifier case. The environment E µ is static (i.e. T = 1), and it contains all images with ground truth detections. For simplicity, we consider one object of interest per image (indexed by i). The observation in this case is a rendered image of an instance of object class i, where the environment parameter µ determines the 3D scene and how the image is rendered (e.g. the camera position/viewpoint, lighting directions, textures, etc.). Here, the observation is defined as the rendering function o 1 : [µ min , µ max ] d → R n . We use Blender [7] to render the 3D scene containing the object and to determine its ground truth bounding box location in the rendered image. The action a 1 by the agent/detector is simply the highest confidence score l i corresponding to class i from the top J detected boxes in o 1 . The final score of F is Q(F, E µ ) = l i . The attack on F is considered successful, if l i ≤ .

B.3. Semantic Adversarial Attack on Autonomous Agents
The semantic adversarial attack of an autonomous agent can also be represented in the general formulation of Algorithm 1 in the paper. Here, A corresponds to the navigation policy, which interacts with a parametrized environment E µ . The environment parameter µ ∈ R d comprises d variables describing the weather/road conditions, camera pose, environment layout etc. In this case, an observation o t is an image as seen from the camera view of the agent at time t. The action a t produced by the navigation policy is the set of control commands (e.g. gas and steering for a car or throttle, pitch, roll and yaw for a UAV). The reward function R(s t , a t ) measures if the agent successfully completes its task (e.g. 1 if it safely reaches the target position at time t and 0 otherwise). The episode ends when either the agent completes its task or the maximum number of iterations T is exceeded. Since the reward is binary, the Q score is the average reward over a certain number of runs (five in our case).

C. Boosting Strategy for BBGAN C.1. Intuition for Boosting
Inspired by the classical Adaboost meta-algorithm [16], we use a boosting strategy to improve the performance of our BBGAN trained in Section 5 of our paper with results reported in Table 1. The boosting strategy of BBGAN is simply utilizing the information learned from one BBGAN by another BBGAN in a sequential manner. The intuition is that the main computational burden in training the BBGAN is not the GAN training but computing the agent A episodes (which can take multiple hours per episode in the case of the self-driving experiments).

C.2. Description of Boosting for BBGANs
We propose to utilize the generator to generate samples that can be used by the next BBGAN. We start by creating the set Ω 0 of the first stage adversary G 0 . We then simply add the generated parameters µ along with their computed scores Q to the training data of the next stage BBGAN (i.e. BBGAN-1). We start the next stage by inducing a new induced set S 1 µ (that may include part or all the previous stage induced set S 0 µ ). However, the aim is to put more emphasis on samples that were generated in the previous stage. Hence, the inducer in the next stage can just randomly sample N points, compute their Q scores and add β * N generated samples from BBGAN-0 to the N random samples. The entire set is then sorted based on the Q scores, where the lowest-scoring s 1 points that satisfy Eq (8) are picked as the induced set S 1 µ , s 1 = S 1 µ . The BBGAN-1 is then trained according to Eq (7). Here β is the boosting rate of our boosting strategy which dictates how much emphasis is put on the previous stage (exploitation ratio) when training the next stage. The whole boosting strategy can be repeated more than once. The global set Ω of all N sampled points and the update rule from one stage to another is described by the following two equations: These global sets Ω k constitute the basis from which the inducer produces the induced sets S k µ . The adversary G k of boosting stage k uses this induced set when training according to the BBGAN objective in Eq (7). Algorithm 2 summarizes the boosting meta-algorithm for BBGAN.

C.3. Empirical proof for BBGAN Boosting
Here we want to show the effectiveness of boosting (Algorithm 2) on improving the performance of BBGAN from one stage to another. Explicitly, we want to show that the following statement holds, under some conditions.

Algorithm 2: Boosting Strategy for BBGAN
Requires: environment Eµ, Agent A number of boasting stages K, boosting rate β, initial training size N Sample N points to form Ω0 like in Eq (19) induce S 0 µ from Ω0 learn adversary G0 according to Eq (7) for i ← 1 to K do update boosted training set Ωi from Ωi−1 as in Eq (20) obtain S i µ from Ωi train adversary Gi as in Eq (7) end Returns: last adversary GK This statement says that the expected score Q of the sampled parameters µ k from the adversary G k of stage (k) BB-GAN is bounded above by the score of the previous stage , which indicates iterative improvement of the fooling adversary G k by lowering the score of the agent A and hence achieving a better objective at the following realxation of Eq (13).

Proof of Eq (21) .
Let's start by sampling random N points as our initial Ω 0 set as in Eq (19) and then learn BBGAN of the first stage and continue boosting as in Algorithm 2. Assume the following assumption holds, then by comparing the average score Q of the entire global set Ω k at stage k (denoted simply as Q(Ω k ))) with the average score of the added boosting samples from the previous stage {µ j ∼ G k−1 } βN j=1 as in Eq (20) (denoted simply as Q(G k−1 )) , two possibilities emerge: 1. Exploration possibility: Q(Ω k )) ≤ Q(G k−1 ) . This possibility indicates that there is at least one new sample in the global set Ω k that are not inherited from the previous stage adversary G k−1 , which is strictly better then G k−1 samples with strictly lower Q score. If the assumption in Eq (23)holds, then the induced set S k µ will include at least one new parameter that is not inherited from previous stage and hence the average score of the induced set will be less than that of the generated by previous stage.
However since the BBGAN of stage k uses the induced set S k µ for training , we expect the samples to be correlated: G k ∼ S k µ , and the scores to be similar as follows: Substituting Eq (25) in Eq (24) results in the inequality which makes the less strict inequality Eq (21)holds . 2. Exploitation possibility: Q(Ω k )) > Q(G k−1 ) . In this scenario, we don't know for sure whether there is a new sample in Ω k that is better than the inherited samples, but in the worst case scenario we will get no new sample with lower score. In either case, the assumption in Eq (23) ensures that the new induced set S k µ is exactly the inherited samples from G k−1 and the following holds.
Using the same argument as in Eq (25), we deduce that in this exploitation scenario Eq (21) is still satisfied. Hence, we prove that Eq (21) holds given the assumption in Eq (23).

C.4. Experimental Details for Boosting
We note that low β values do not affect the training of our BBGAN since the induced set will generally be the same. Hence, we use β = 0.5, a high boosting rate. For practical reasons (computing 50% of the training data per boosting stage is expensive) we just compute 10% of the generated data and repeat it 5 times. This helps to stabilize the BB-GAN training and forces it to focus more on samples that have low scores without having to evaluate the score function on 50% of the training data. Tables 4 and 5 Figure 7 shows some qualitative examples for each of the 12 object classes. These images were rendered according to parameters generated by BBGAN which fooled the detector.    to 1). Also, a histogram of agent scores for generated parameters and a histogram of scores for random parameters are shown in the right column.

H.1. Diagnosis
To identify weaknesses and cases that results in systematic failure for the YOLOv3 detector, we fix some semantic parameters and attack the others. We conduct two different case studies. The first one involves the camera view-point and light direction; these have the largest impact in determining the pixel values of the final image. The second case study examines occlusion, which is one of the most common causes of detection failure. In both cases, we focus on two classes that are relevant in the autonomous driving application: cars and motorbikes. Since we have several 3D models for each class, we include the effect of model shapes in the analysis. We use roughly sampled models with homogeneous texturing (obtained from the training in Section4.1) as well as detailed fully-textured models. This also has the nice side-effect to show how well our insights transfer to more realistic renderings and ultimately the real world. In total, we consider five different scenarios per case. Scenario 1: simple car, Scenario 2: detailed car, Scenario 3: simple motorbike, Scenario 4: detailed motorbike 1, Scenario 5: detailed motorbike 2. We use these scenarios for both cases.
Case 1: View point. We restrict the number of parameters to 4 (φ cam , θ cam , φ light , φ light ), and fix the object class and RGB colors (pure blue). Case 2: Occlusion. Since occlusion plays an important role in object misdetection, we introduce an occlusion experiment. Here, we investigate how occlusion (e.g. by a pole) can result in failure of a detector (e.g. from which view point). Therefore, we include the camera viewpoint angles (φ cam , θ cam ) and introduce a third parameter to control horizontal shift of a pole that covers 15% of the rendered image and moves from one end to another. The pole keeps a fixed distance to the camera and is placed between the camera and the object.

H.2. Transferability
Across Shape Variations. We use the same scenarios as above to construct transferability experiments. The goal is to validate generalization capabilities of the learned fooling distribution from one z ∅ ∅  Tables 6 and 7 show the transferability of the adversarial attacks for Case 1 and Case 2. We see that most attacks transfer to new scenarios that are similar, indicating the generalization of the learned fooling distribution. However, the attacks that were learned on more detailed CAD models transfer better to generic less detailed models (e.g. PASCAL3D [49] and ShapeNet[10] models).
To validate generalization capabilities of the learned fooling distribution, we learn this distribution from samples taken from one setup and then test it on another. Table 6 shows that adversarial distributions learned from detailed textured models transfer better (i.e. maintain similar AFR after transferring to the new setup) than those learned from rough ones from Pascal3D [49] and ModelNet [48]. This observation is consistent with that of [3].   Table 7: Case 2: occlusion attack transferability. Attack Fooling Rate for sampled attacks on each scenarios and also for transferred attacks from one scenario to another. Random attacks for each scenario are put for reference YOLOv3 detector on these images and observe the confidence score for the 'motorbike' class of interest. On the samples generated from the BBGAN distribution, the attack fooling rate is 21% compared to only 4.3% when picking a random viewpoint. In Figures 19 and 20, we visualize the fooling distribution generated by our BBGAN and provide some corresponding real-world images.
Visualization. In Figures 19 and 20, we visualize of the fooling distribution generated by our BBGAN in the two previous experiments (SectionH.1). We also include some real-world images captured according to the parameters generated by the BBGAN. We fix the object to be a car and fix the distance to the camera and train a BBGAN to learn the fooling camera and light source angles to fool the YOLOV3 detector. Top: on the right we plot the camera positions and light source directions of 250 sampled parameters in a 3D sphere around the object. On the left we show how taking real photos from the same rendered angles of some toy car confuses the YOLOV3 detector as the rendered image. Bottom: on the right we visualize the distribution of parameters normalized from (-1,1), while on the left we visualize the histogram of scores (0 to 1) of the learnd parameters distribution vs random distribution. We fix object to be a mototrbike and fix the distance to the camera and train a BBGAN to learn the fooling camera and light source angles to fool the YOLOV3 detector. Top: on the right we plot the camera positions and light source directions of 250 sampled parameters in a 3D sphere around the object. On the left we show how taking real photos from the same rendered angles of some toy motorbike confuses the YOLOV3 detector as the rendered image. Bottom: on the right we visualize the distribution of parameters normalized from (-1,1), while on the left we visualize the histogram of scores (0 to 1) of the learned parameters distribution vs random distribution. In our YOLOV3 experiments, we consistently found that for most objects top rear or top front views of the object are fooling the YOLOV3 detector. Furthermore, the light angle which will result in highest reflection off the surface of the object also results in higher fooling rates for the detector. The color of the object does not play a big role in fooling the detector, but usually colors that are closer to the background color tend to be preferred by the BBGAN samples (as shown in the qualitative examples). From the analysis in SectionH of transferability of these attacks, we note that attacks on more detailed CAD shapes and models transfer better to less detailed shapes, but the opposite is not true.

I.2. Self-driving cars:
In our experiments we found that weather is the least important parameter for determining success. This is probably due to the fact that the driving policy was trained on multiple weather conditions. This allows for some generalization and robustness to changing weather conditions. However, the driving policy was trained with a fixed camera. We observe, that the driving policy is very sensitive to slight perturbations of the camera pose (height and pitch).

I.3. UAV Autonomous Navigation:
We observe that the UAV fails if the track has very sharp turns. This makes intuitive sense and the results that were produced by our BBGAN consistently produce such tracks. For the tracks that are only parameterized by three control points it is difficult to achieve sharp turns. However, our BBGAN is still able to make the UAV agent fail by placing the racing gates very close to each other, thereby increasing the probability of hitting them.