{'fraction_non_alphanumeric': 0.03128633336856569, 'fraction_numerical': 0.010745869006095197, 'mean_word_length': 5.092294483794806, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception

Recent advancements in Vehicle-to-Everything communication technology have enabled autonomous vehicles to share sensory information to obtain better perception performance. With the rapid growth of autonomous vehicles and intelligent infrastructure, the V2X perception systems will soon be deployed at scale, which raises a safety-critical question: how can we evaluate and improve its performance under challenging traffic scenarios before the real-world deployment? Collecting diverse large-scale real-world test scenes seems to be the most straightforward solution, but it is expensive and time-consuming, and the collections can only cover limited scenarios. To this end, we propose the first open adversarial scene generator V2XP-ASG that can produce realistic, challenging scenes for modern LiDAR-based multi-agent perception system. V2XP-ASG learns to construct an adversarial collaboration graph and simultaneously perturb multiple agents' poses in an adversarial and plausible manner. The experiments demonstrate that V2XP-ASG can effectively identify challenging scenes for a large range of V2X perception systems. Meanwhile, by training on the limited number of generated challenging scenes, the accuracy of V2X perception systems can be further improved by 12.3% on challenging and 4% on normal scenes.


I. INTRODUCTION
Over the past decade, we have witnessed tremendous progress in Autonomous Vehicles (AVs) [1], [2] and intelligent transportation systems [3]. Sooner or later, these autonomous systems will be deployed on roads at scale, opening up opportunities for cooperation between them. Previous works in [4], [5], [6], [7], [8], [9], [10], [11], [12] have demonstrated that by leveraging the Vehicle-to-Everything (V2X) communication technology, AVs and infrastructure can perform cooperative perception by using the shared sensing information and thus significantly enhance the perception performance. Despite the remarkable improvement, these works evaluate the proposed systems on the dataset with natural scenarios that do not contain sufficient safety-critical scenes. Under the challenging scenes, these systems may have inferior performance, and thus it is crucial to identify challenging scenes to fully understand the robustness of existing cooperative perception systems. The straightforward solution is to collect a wide range of testing scenes in the real world and identify critical ones. However, compared to single-agent systems, the cost and time consumption of gathering and labeling data for multiagent systems can be much more demanding. A preferable cost-effective solution is to generate large-scale realistic scenes [13], [14], [15] in high-fidelity simulators. Yet these approaches only consider the common scenes and lack the capability of performing stress tests on corner cases for the target systems.
To address this problem, inspired by recent works [16], [17], [18] that produce safety-critical scenarios for the single-agent planner by perturbing surrounding vehicles' trajectories, we aim to automatically generate diverse and challenging scenes for V2X perception systems. Nonetheless, it is demanding to directly transfer the single-agent scenario generation methods to V2X systems. Unlike the single-agent system with a single viewpoint, V2X system involves collaborators with multiple viewpoints. The geometric relationships between these views from distinct collaborators can vastly influence the perception performance [19], which is not considered in the previous scene generators. Furthermore, the collaborator choices can influence the viewpoints and thus are coupled with the agent pose perturbations, leading to more complicated interactions. Handling the differences mentioned above between singleagent and cooperative perception is the key to an effective V2X adversarial scene generator. However, no literature has been reported to investigate them. Moreover, most existing methods are tailored for planning, and there is no publicly available scene generation framework for perception, which hinders the progress of developing efficient scene generation methods for modern single-agent and V2X perception systems.
To this end, we introduce V2XP-ASG -the first open Adversarial Scene Generator for LiDAR-based V2X Perception systems. To the best of our knowledge, this is the first framework that aims to automatically generate challenging scenes for V2X perception. As shown in Figure 1, given an initial scene from the dataset, V2XP-ASG first constructs an adversarial collaboration graph by searching for adversarial collaborators whose viewpoints combination will lead to inferior performance. Afterward, we perturb multiple agents' poses in an adversarial and plausible way. To reflect the updated LiDAR observation caused by sensor viewpoint change and pose perturbation, we conduct the experiments based on the high-fidelity CARLA [20] simulator. Extensive experiments show that V2XP-ASG can create challenging scenes to severely deteriorate the performance of modern V2X perception systems. More importantly, training with these adversarial scenes can increase the system's accuracy. Our main contributions can be summarized as follows • We present V2XP-ASG, the first open adversarial scene generation framework, to test modern V2X perception systems. Our V2XP-ASG can intelligently generate challenging scenarios for a wide range of V2X perception systems plausibly. We will release codes in the future.
• We formulate this scene generation task as a novel twostage optimization problem, which makes the optimization easier and provides interpretability to the system's weakness. • We propose an efficient search strategy for the novel Adversarial Collaborator Search (ACS) task in the first stage by leveraging learnable collaboration graph weights. Moreover, in the second stage, we employ the black-box optimization algorithms with customized adversarial objectives for perception tasks to perturb vehicle poses in an adversarial and feasible manner. These two stages are combined to efficiently raise the scene's challenging level. • As demonstrated by our experiments, training on generated adversarial scenes can greatly improve the system's precision under both normal and challenging scenes.

II. RELATED WORK
V2X Perception: V2X perception investigates how to leverage the visual information from nearby AVs and intelligent infrastructure to enhance the perception capability. Based on the collaboration strategies, there are three major classes: early [21], late [22], [23], [24], and intermediate fusion [5], [7], [10], [4], [25], [26], [27]. The early fusion method delivers the raw point clouds across agents, and each agent will feed the aggregated point clouds to the network for 3D detection. Despite preserving complete raw information, early fusion usually requires large bandwidth, making it unrealistic to deploy [28], [29]. On the contrary, late fusion has a minimum data transmission size as it only circulates the metadata of prediction outputs. However, its accuracy is limited since it fails to provide valuable scenario context [5]. To achieve a good trade-off between bandwidth and accuracy, intermediate fusion, which broadcasts the compressed intermediate neural features, has been mostly studied recently. [5] proposes a spatial-aware graph neural network for joint perception and prediction, and [4] employs knowledge distillation to advance the learning with the supervision of early fusion.
[7] proposes a location-wise self-attention mechanism to fuse the features from different AVs. This work evaluates all three fusion strategies and the single-agent perception method.

Adversarial Scenario Generation:
Recently several works [16], [17], [18], [30] have been proposed to generate safety-critical scenarios for identifying planning failures. [16] uses Bayes Optimization to search crash scenarios based on a hierarchical graph route in CARLA [20]. STRIVE [17] represents agents in latent space via a graph-based conditional VAE model and perturb latent variables via gradient-based optimization to produce trajectories that collide with a given planner. AdvSim [18] benchmarks several black-box optimization algorithms to search adversarial trajectories to obtain safety-critical scenarios for the full autonomy stack, but its adversarial objective is targeted for planning, and it is designed for the single vehicle system. Another stream of work [31], [32] formulates the scenario generation problem as the rare event simulation to sample failure scenarios for the single-agent autonomy system. In contrast, we focus on producing challenging scenarios for the LiDAR-based multiagent V2X perception system where both the agents' poses and the selection of collaborators are searched to optimize an adversarial objective customized for perception. LiDAR-based adversarial attack: LiDAR-based adversarial attacks [33], [34], [35] in a physically realizable way have gained increasing attention. [35] performs the first security study of LiDAR-based perception in AV settings by changing raw LiDAR points via a LiDAR spoofer. [34] studies the backdoor of motion compensation and performs adversarial spoofing of the agent's trajectory to deteriorate the perception performance. In [33], an adversarial mesh is put on the rooftop of a vehicle, and mesh poses are optimized to make the vehicle invisible. Besides the above single-agent attack, [36] attacks the V2X perception system by adding adversarial noises to the intermediate features shared by intelligent agents. Different from previous work, we directly attack the agent collaboration choice and vehicle poses to deteriorate the performance of the modern V2X perception system, which ensures the feasibility and plausibility of the generated adversarial examples. Multi-agent collaboration graph: A classical multi-agent collaboration graph focuses on fusing information from selected agents or all the connected agents to increase the system's performance. Who2com [19] proposes a handshake communication mechanism where the neural network can learn and determine which two agents should compress relevant information needed for each stage. When2com [37] constructed learning-based communication groups to learn to communicate and used an asymmetric attention mechanism to decide when to communicate across a fully-connected graph. DiscoGraph [4] exploited a method that incorporates both early and intermediate collaboration into a knowledge distillation framework, which enables the knowledge of early collaboration to guide the training of an intermediate fusion model. However, rather than trying to improve the perception performance by finding robust collaborators, in this work, we are interested in searching for adversarial collaborators to deteriorate the task performance, which can help expose the potential vulnerability of the multi-agent system.

III. ADVERSARIAL SCENE GENERATION
V2XP-ASG aims to generate realistic, challenging scenes for a given V2X perception model. The overall architecture of our framework is shown in Figure. 1, which consists of Adversarial Collaborator Search (ACS) and Adversarial Perturbation Search (APS). Given an existing scene with a fixed ego agent, we first search for adversarial collaborator combinations by leveraging an attention-based sampling approach and then build the adversarial collaboration graph with selected agents. Afterward, we perturb multiple agents' poses and feed updated LiDAR observation into the target perception system to generate 3D bounding boxes. Eventually, we evaluate the predictions with adversarial objectives and perform black-box optimization to update poses.

A. Problem formulation
Given an initial scene with a set of agents A = {a 1 , . . . , a N } where agents a i can be either vehicle or infrastructure, we assume k of them are equipped with sensing and communication devices and are denoted as intelligent agents I ⊂ A. Within the communication range, all the agents belonging to I can share information with each other and one of them is selected as the ego agent to aggregate the sensing information to form a unified detection. More formally, the set of observations from these agents are (1) where S = {s 1 , . . . , s N } is the set of poses of all agents in the scene. X i (S) is the sensing observation from intelligent agent i and it reflects the change of agents' poses S. The ego agent with neighboring collaborators will collaboratively predict the bounding boxesŶ = f (X (S, I) ; τ ) based on the aggregated sensing observations X , where f is the V2X 3D detection model, and τ is the fusion strategy including early , late and intermediate fusion.
Our objective is to attack the V2X perception model by adjusting initial normal scenes to be more challenging. There are two types of adversaries in this work, namely agent collaboration combination I and pose perturbation ∆. These two adversaries will modify the sensor viewpoint and vehicle poses in a physically plausible way and CARLA [20] is leveraged to simulate the updated observations X (S + ∆, I).
We then define the adversarial objective L adv (Section III-D) which is minimized to generate challenging scenes: where | · | is the set cardinality and Y is the ground truth bounding boxes. We decouple the above formula and optimize this factorized approximation as follows: Such factorization can separate optimization into two subsequent tasks: adversarial collaborator search (Eq. 3) and adversarial pose perturbation (Eq. 4), which can not only simplify the optimization but also provides the possibility of analyzing two stages individually to better understand the performance of the target system. As the performance drop brought by pose perturbation can be easily diminished by changing sensor viewpoints, we choose to search collaboration first instead of the reverse order. For multi-agent systems, k = 1 will clearly lead to inferior performance. However, we are interested in studying the robustness of V2X system even when it is deployed at scale. Thus we fix k = 3 in this work.

B. Adversarial Collaborator Search
Although examining all the agent combinations may seem appealing at first but it is prohibitively expensive due to the enormous combinations and the high computation cost. Thus instead, we design an efficient search algorithm. We adopt the intermediate fusion model AttFuse [7] to construct the collaboration graph where the learnable edge weights are considered to represent the importance of that agent's feature contribution to the ego agent. We leverage these learnable edge weights to define each agent's weakness level so that lower values represent smaller contributions to the overall perception system and thus, assigning these weak agents with higher sampling probability can increase the chance of finding adversarial collaborators with inferior performance. Although other fusion methods such as Early Fusion and Late Fusion can't produce collaboration graphs with important weights, we experimentally demonstrate that the proposed attentionbased probabilistic sampling method can be transferred to other models with great search efficiency ( IV). In addition, our framework is general and other adversarial collaborator search methods can be integrated into the framework and we encourage more researchers to investigate this new direction.
To obtain the edge weights for all agents, we first equip all agents with LiDAR sensor in the simulator. Each agent will reason intermediate features based on its raw sensing observation and then transmit their intermediate features H i ∈ R H×W ×C to the ego agent. Then the self-attention model is employed to capture the feature importance from different agents in the same spatial locations under ego coordinate. Formally, let h i mn = [H i ] mn ∈ R C be the feature from agent i in location (m, n) and h mn = {h 1 mn , . . . , h N mn } ∈ R N ×C is the aggregated features for location (m, n) from all N agents. Then the attention is operated as follows: where q mn , k mn and v mn are linear projections of h mn along channel dimension. The fused feature h mn = a mn v mn . It can be sliced and rearranged to get the updated feature H i . Before applying this self-attention model in our scene generator, we will train it on the augmented OPV2V dataset [7] to learn the collaboration graph construction by sending H i to a detection header to produce bounding box predictions. During the adversarial procedure, we fix the network's weights and directly exploit calculated a mn to detect the adversarial collaborators. As the attention weights are position-wise, average pooling is adopted to aggregate the weights of all spatial locations.
Here s j represents the importance of agent j and thus 1/s j is the associated weakness level for this agent. For each collaborator combination I, we can model its weakness level as the sum of the individual agent's weakness level and apply softmax to form a probability distribution: where τ is the temperature parameter to control the shape of the distribution. We sample k 0 combinations without replacement according to the probability p, examine all the adversarial objective values, and keep the best one with the lowest adversarial value as the adversarial collaboration combination I * . This combination is eventually used to build the adversarial collaboration graph where only the selected agents can share the information with each other.

C. Adversarial Perturbation Search
Search space: We perturb multiple agents' poses within physical plausible bounds. Each agent's perturbation is Generate ∆ (i) based on historical observation D and black-box search algorithm 6: Project ∆ (i) onto feasible set Q where m is the number of perturbed agents and these m agents are sampled according to an occlusion-inspired heuristic from the set A. We rank N agents by occlusion levels in decreasing order and then select the top m agents to perturb. Specifically, each agent's occlusion level is the sum of its intrinsicocclusion score and extrinsic-occlusion score. The intrinsicocclusion score is 1 if the agent is partially overlapped by any other agent in the scene, indicating the occlusion level of the perturbed agent (occludee). The extrinsic-occlusion score measures how many other agents in the scene can be partially occluded by the examined agent (occluder). For the multiagent V2X system, we sum the occlusion score from different viewpoints as the agent's overall score. In this way, the sampled agents are more likely to observe partially occluded objects. We set ∆ ≤ δ to constrain the perturbation within a limited range. In this work, m = 3 is adopted for increased search space and scenario configurations. Black-box search algorithm: Our framework is amenable to any black-box search algorithm. The search algorithm seeks to find challenging scenes for the V2X perception model by minimizing the adversarial objective L adv . In this paper, we benchmark 3 black-box search algorithms: Random Search (RS), Genetic Algorithm (GA) [38] and Bayesian Optimization (BO) [39]. The Random Search will randomly sample perturbation from the feasible set. The Genetic Algorithm will maintain a population of candidate perturbations, evolve the population over time by selecting parents with high fitness scores, and the best candidate is always kept for enhanced performance [38]. The Bayesian Optimization will build a surrogate model and use the acquisition function to balance the exploration and exploitation for generating candidates.
Search pipeline: The overall method is summarized in Algorithm 1. To increase the plausibility of the perturbation, we build a context-aware feasible perturbation set Q for the sampled m agents. To generate such a feasible set, we first uniformly sample N Q = 1000 potential perturbations within the bounds (δx, δy ∈ [−2.5, 2.5] m and δθ ∈ [−45 • , 45 • ]) and remove the ones that can cause potential collisions. As the location and angle have different magnitudes, to diminish this bias, we also normalize the perturbation ∆ by dividing it by the perturbation range s.t. ∆ ∞ ≤ 1. A historical observation D of (∆, L adv ) pairs is maintained through the optimization process. During each iteration, the search algorithm generates the potential perturbation ∆ and projects it onto the feasible set Q by finding the closest element in the set measured in 2 distance. Then it will query the target V2X perception model to obtain adversarial loss L adv . The search algorithm will update the historical observation D, which is used to generate the perturbation for the next iteration. The best-observed perturbation with the lowest adversarial loss is selected for the final perturbation.

D. Adversarial Objective
We adopt weighted average precision as the adversarial loss.
where T is a set of Intersection of Union (IoU) thresholds and in our experiment, T = {0.3, 0.5, 0.7}. AP@t is the average precision (AP) at IoU of threshold t and w t is the associated weight. Using this weighted sum of average precision can lead to smoother score change, which eases the difficulty of optimization. In this work, we set w 0.3 = 1, w 0.5 = 0.8 and w 0.7 = 0.5 to give higher weights for lower IoU thresholds as their APs are generally harder to decrease.

IV. EXPERIMENTS A. Experiment Setup
Dataset: Experiments are conducted in CARLA [20] simulator. Two datasets are used in the experiment: 1) we augment the existing large-scale Vehicle-to-Vehicle (V2V) perception dataset, namely OPV2V, with infrastructure sensor data. The OPV2V's training split is used to train modern V2X perception models until convergence. From the augmented OPV2V's test split, we have 94 driving logs and each has 70 frames, covering 8 road areas. Due to the high computational cost of running the CARLA simulator and deep networks, We sample 331 Normal scenes from these collected scenarios, and the train/validation splits for evaluating V2XP-ASG are 219/112. The major experiments are conducted in train split and the validation split is only used for reporting the performance of the fine-tuned model. 2) Besides the above data, we build another heuristic-generated challenging dataset called Heuristic that has high traffic density with severe occlusions and sparse observations. This hold-out dataset includes 321 frames sampled from 14 driving logs. We first collect 14 driving logs for a total of 980 frames with dense traffic (20∼30 vehicles per frame) in 4 CARLA towns. Then for each frame, we calculate the average number of LiDAR points within the bounding boxes, and we curate the raw data by only keeping the scene with an average number of points less than the threshold N thred = 25. As this dataset is only visible during inference, it provides a fairer evaluation of whether the fine-tuned model can achieve better results in unseen challenging scenes (Table II). V2X perception model: V2XP-ASG is evaluated on 4 models: 1) No Fusion, which predicts bounding boxes only based on ego-agent LiDAR point clouds. 2) Late Fusion, where detection proposals of each agent are transmitted and the non-maximum suppression is applied to generate the final predictions. 3) Early Fusion, which aggregates raw LiDAR points from different agents to a holistic view and feeds them to the detector. 4) AttFuse [7], an intermediate fusion model which broadcasts intermediate features with each other and uses location-wise self-attention to fuse received features. All the models are implemented with PointPillar [40] backbone. V2XP-ASG: We adopt AttFuse's attention weights for constructing the adversarial collaboration graph for all 3 V2X perception models. The temperature parameter τ = 0.03 and we examine k 0 = 3 combinations sampled according to Eq. 8. For the main experiments, we adopt the Bayesian Optimization [39] as the black-box search algorithm. For LiDAR simulations, to diminish the influence of noise, we set the noise to be zero i.e., zero drop-off rate and zero noise level so that the perception performance is solely affected by the intrinsic scene configurations. The perception performance is evaluated in the range of x, y ∈ [−48, 48] m. For consistent ground truth, when perturbing agents, we ensure that the updated poses still stay within the evaluation range. Parameters of black-box optimization can be found in the supplementary videos.

V2X perception model evaluations:
The evaluation results for different V2X perception models are shown in Table I. The results demonstrate that our V2XP-ASG can generate challenging scenes for both single-agent and V2X perception systems with an average of 28.7% drop of AP@0.7, showing that V2XP-ASG can efficiently identify challenging scenes for a wide range of models with different fusion strategies.
Improvement with challenging scenes: We now investigate if our generated challenging scenes can help improve the performance of the V2X perception system. We first finetune the original AttFuse model on train split of its generated challenging scenes and then test the model on the val split of Normal and Challenging scenes. For a fairer comparison, we      also evaluate its performance on a separate challenging holdout set (Heuristic). As shown in Table II, the fine-tuned model show improvement for all 3 datasets, illustrating the great benefit of V2XP-ASG for improving perception systems. Transferability of challenging scenes: Table III shows the transferability of V2XP-ASG generated scenes. The source model is used for generating challenging scenes and the target model is tested on these generated scenes. The performance is the best when testing the target model on the scenes generated by the same model. Moreover, three models all show great performance drop on each other's generated challenging scenes than their performance on normal scenes, showing that the models have a certain degree of agreement on whether a scene is challenging and thus demonstrating the favored transferability of V2XP-ASG generated challenging scenes.
Baseline comparison: We compare the proposed AttFusebased collaborator selection strategy (ACS-A) with random selection (ACS-R) that randomly sample k 0 = 3 agent combinations. The experiment results are shown in Table IV. The proposed method can decrease the detection performance by 9.5%, 8.9%, 7.7% while random selection can only decrease the score by 6.5%, 6.8%, 4.1%, showing great efficiency of the proposed search strategy.
Component analysis: As shown in Table V, the proposed two components ACS and APS are both instrumental for find-ing challenging scenes, demonstrating that both viewpoints and pose perturbation are critical for identifying challenging scenes for the multi-agent system.
Black-box search benchmark results: The benchmark results for different black-box search strategies are shown in Table VI. Both GA and BO outperform RS and BO reaches the best performance. We argue this is due to the good balance between exploration and exploitation of BO.
Visualization results: Figure 2 shows the visualization of V2XP-ASG generated Challenging scenes and the associated Normal scenes. As shown in Figure 2b, adversarial collaborator selection can change sensor viewpoint to make objects hard to detect. In Figure 2c, three agents are perturbed and two of the perturbed agents simultaneously take a right turn, which is realistic and rare in real world. It demonstrates that the V2XP-ASG generated scenes are both challenging and meaningful for testing the perception module.

V. CONCLUSIONS
In this work, we present the first open adversarial scene generation framework dubbed V2XP-ASG for LiDAR-based V2X perception systems. The experiments demonstrate that the V2XP-ASG can create challenging scenes for a wide range of perception models, including single-agent and V2X perception systems. In particular, by training on generated scenes, the performance of the perception system can be further improved for both common and challenging scenes. Moreover, the ablation study verifies that the proposed attention-based probabilistic sampling approach can effectively find adversarial collaborators and the two novel components are both advantageous for generating challenging scenes.