{'fraction_non_alphanumeric': 0.04396779020879356, 'fraction_numerical': 0.015249768403049953, 'mean_word_length': 5.0601698574924425, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Common Corruption Robustness of Point Cloud Detectors: Benchmark and Enhancement

Object detection through LiDAR-based point cloud has recently been important in autonomous driving. Although achieving high accuracy on public benchmarks, the state-of-the-art detectors may still go wrong and cause a heavy loss due to the widespread corruptions in the real world like rain, snow, sensor noise, etc. Nevertheless, there is a lack of a large-scale dataset covering diverse scenes and realistic corruption types with different severities to develop practical and robust point cloud detectors, which is challenging due to the heavy collection costs. To alleviate the challenge and start the first step for robust point cloud detection, we propose the physical-aware simulation methods to generate degraded point clouds under different real-world common corruptions. Then, for the first attempt, we construct a benchmark based on the physical-aware common corruptions for point cloud detectors, which contains a total of 1,122,150 examples covering 7,481 scenes, 25 common corruption types, and 6 severities. With such a novel benchmark, we conduct extensive empirical studies on 8 state-of-the-art detectors that contain 6 different detection frameworks. Thus we get several insight observations revealing the vulnerabilities of the detectors and indicating the enhancement directions. Moreover, we further study the effectiveness of existing robustness enhancement methods based on data augmentation and data denoising. The benchmark can potentially be a new platform for evaluating point cloud detectors, opening a door for developing novel robustness enhancement methods.


I. INTRODUCTION
O Bject detection via LiDAR-based point cloud [1], [2], as a crucial task in 3D computer vision, has been widely used in applications like autonomous driving [3]. Recently, the data-driven methods (i.e., deep neural networks) have significantly improved the performance of 3D point cloud detectors [4], [5], [2] on various public benchmarks, e.g., KITTI [6], NuScenes [7], and Waymo [8]. However, the scenarios covered by these public benchmarks are usually limited. For instance, there is a lack of natural fog effects in these datasets, while fog could affect the reflection of laser beams and corrupt point cloud data with false reflections by droplets [9], [10]. Apart from the external scenarios, the internal noise of sensors can also increase the deviation and variance of ranging measurements [11] and result in corrupted data and detector performance degradation. Given that LiDAR-based point cloud detection is usually used in safety-critical applications (e.g., autonomous driving) and these external and internal corruptions could potentially affect detectors' robustness [12], [13], [11], it is critical to comprehensively evaluate an object detector under those corruptions before deploying it in real-world environments.
There are some works constructing datasets while considering extreme weather like CADC [14], Boreas [15], SeeThrough-Fog (STF) [10]. Nevertheless, the constructed datasets only consider limited situations in the real world due to the heavy collection costs, which are far from a comprehensive evaluation. For instance, Boreas only covers 4 rainy scenes and 5 snowy scenes. STF only contains foggy point clouds at severity levels of "dense" and "light". Hence, there is an increasing demand for extending existing benchmarks to conduct a comprehensive evaluation through covering diverse corruptions in the real world. A straightforward way is to synthesize the corrupted point clouds given the success of similar solutions in the imagebased tasks [16], [17] and 3D object recognition [11], [18]. However, there is no accessible dataset for the robustness evaluation of point cloud detectors. Note that, the robustness datasets (e.g., Modelnet40-C [11]) for 3D object recognition cannot be used to evaluate the point cloud detectors, directly: (1) the example in the recognition dataset only contains the points of an object and cannot be adopted for object detection task that aims to localize and classify objects in 3D scene. (2) The latest Modelnet-C [18] and Modelnet40-C [11]) only consider 7 corruptions and 15 corruptions, respectively, which is still limited for a comprehensive evaluation in safety-critical environments such as autonomous driving.
The main challenge for building a dataset for the robustness evaluation of point cloud detection stems from the huge amount of diverse corruption types with different physical imaging principles. For example, flawed sensors and different object characteristics could lead to noise-like corruptions and affect spherical and Cartesian coordinates of points, respectively. Different weathers like rain and fog might lead to false reflections. These corruptions have different imaging principles and need careful designs of the respective simulation methods.
In this work, for the first attempt, we construct a benchmark to evaluate the robustness of point cloud object detectors based on LiDAR under diverse common corruptions and discuss the effectiveness of existing robustness enhancement methods. Regarding the benchmark construction, we first design physicalaware simulation methods for 25 corruptions according to their physical models, respectively. Then, we borrow 7,481 raw 3D scenes (i.e., clean point clouds) from [6] and build large-scale corrupted datasets by adding 25 corruptions with 6 different severity levels to each clean point cloud. Finally, we obtain a total of 1,122,150 examples covering 7,481 scenes, 25 common corruption types, and 6 severity levels. Compared with real-world data benchmark (see Table I), the proposed benchmark synthesized more examples for benchmarking robustness. Compared with other synthesized benchmark (see Table I), our benchmark provides more types of corruption patterns to specifically support benchmarking object detection. Note that, we conduct extensive experiments to quantitatively validate the effectiveness of simulation methods by evaluating the naturalness of synthesized data. With such a novel benchmark, we investigate the robustness of current point cloud detectors by conducting extensive empirical studies on 8 existing detectors, covering 3 different representations and 2 different proposal architectures. In particular, we study the following four research questions to identify the challenges and potential opportunities for building robust point cloud detectors: • How do the common corruption patterns affect the point cloud detector's performance? Given overall common corruptions, an accuracy drop of 11.01% (on average) on all detectors anticipates a noticeable accuracy drop of detectors against diverse corruption patterns. • How does the design of a point cloud detector affect its robustness against corruption patterns? Compared with two-stage detectors, one-stage detectors perform more robust against a majority of corruptions. Compared with point-based detectors, voxel-involving detectors perform more robust against the most of corruptions. • What kind of detection bugs exist in point cloud detectors against common corruption patterns? Followed by the decrease in the rate of true detection, common corruptions widely trigger a number of false detections on all point cloud detectors. • How do the robustness enhancement techniques im-prove point cloud detectors against common corruption patterns? Even with the help of data augmentation and denoising, common corruptions still cause a severe accuracy drop of over 10% on detection. In summary, this work makes the following contributions: • We design physical-aware simulation methods covering 25 common corruptions related to natural weather, noise disturbance, density change, and object transformations at the object and scene level. • We create the first robustness benchmark of point cloud detection against common corruptions. • Based on the benchmark, we conduct extensive empirical studies to evaluate the robustness of 8 existing detectors to reveal the vulnerabilities of the detectors under common corruptions. • We study the existing data augmentation (DA) method and denoising method's performance on robustness enhancement for point cloud detection and further discuss their limitations.

A. LiDAR Perception
LiDAR perception is sensitive to both internal and external factors that could result in different corruptions. Adversarial weather [9] (e.g., snow, rain, and fog) can dim or even block transmissions of lasers by dense liquid or solid droplets. Regarding noise characteristics of point clouds, strong illumination [22] affects the signal transmission by lowering Signal-to-Noise Ratio (SNR), increasing the noise level of LiDAR ranging [23]. Besides, the intrinsically inaccurately ranging and the sensor vibration [24], [25] potentially trigger noisy observations during LiDAR scanning. Environmental floating particles (e.g., dust [26]) could perturb point cloud with the background noise. Density distribution of LiDAR-based point clouds can also easily affect autonomous driving. For instance, common objectobject occlusions block LiDAR scanning on objects in the scene [13]. Besides, the dark-color cover and rough surface Density cutout Sensor: different scanning layers, object occlusion [13], and randomly laser beam [13] or layer (rotary laser) malfunction; rotation local_dec shear Object deformation: bending or moving pedestrians [28], different styles of vehicles [29]. local_inc FFD beam_del scale layer_del [21] could affect LiDAR's reflection and thus reduce local point density when sensing such objects. Moreover, the malfunction of (fixed or rotary) lasers [13] globally loses points or layers of points in point clouds. For 3D tasks, various shapes [28], [29], locations and poses [27] of objects can also influence the context perception in the scene. Apart from these natural corruptions, LiDAR perception is also sensitive to adversarial attack. Adversarial attacks [30] pose significant security issues and vulnerability on 3D point cloud tasks (e.g., classification [31], detection [32], and segmentation [33]).

C. Robustness Benchmarks against Common Corruptions
Several attempts have been made to benchmark robustness for different data domains. Based on ImageNet [42], ImageNet-C simulates real-world corruptions to test image classifiers' robustness. ObjectNet [17] illustrates the performance degradation of 2D recognition models considering object backgrounds, rotations, and imaging viewpoints. Inspired by 2D works, several benchmarks were built for 3D tasks. Modelnet40-C [11] corrupts ModelNet40 [43] with 15 simulated common corruptions affecting point clouds' noise, density, and transformations, to evaluate the robustness of point cloud recognition. Targeting 7 fundamental corruptions (i.e., "Jitter", "Drop Global/Local", "Add Global/Local", "Scale", and "Rotate"), ModelNet-C reveals the vulnerability of different components of 9 existing point cloud classifiers. Regarding point cloud detection, NuScenes, Waymo, and STF collect LiDAR scans under adversarial rainy, snowy, and foggy conditions, where the accuracy of 3D detectors is tested [7], [10], [8]. However, to the best of our knowledge, a lack of benchmark of point cloud detection's robustness comprehensively against various common corruptions is still remaining.

D. Robustness Enhancement for Point Cloud Detection
Recently, improving the robustness of point cloud detection has also received significant concerns. Zhang et al. propose PointCutMix [44] as a single way to generate new training data by replacing the points in one sample with their optimal assigned pairs in another sample. Lee et al. [45] propose a rigid subset mix (RSMix) augmentation to get a virtual mixed sample by replacing part of the sample with shape-preserved subsets from another sample. Specifically for 3D object detection, there are several ways to improve detectors' robustness. Choi et al. [46] propose a part-aware data augmentation that stochastically augments the partitions of objects by 5 basic augmentation methods. LiDAR-Aug [47] presents a rendering-based LiDAR augmentation framework to improve the robustness of 3D object detectors. LiDAR light scattering augmentation [12] and LiDAR fog stimulation [48] utilize physics-based simulators to generate data corrupted by fog/snow/rain and then augment object detectors. Self-supervised pre-training [49], [50] can also endow the model with resistance to augmentation-related transformations. Besides, denoising methods [51], [52], [53] can remove the outliers in point clouds and thus potentially improve detectors' robustness. Regarding module design, there are also some detectors specialized for resisting corruptions, e.g. BtcDet [13] with the occupancy estimator for estimating occluded regions and Centerpoint [54] with key-point detector for a flexible orientation regression. In this paper, we evaluate part-aware data augmentation and K-nearest-neighbors-based filtering methods for improving point cloud detectors against diverse common corruption patterns.

A. Point Cloud Detection
Point clouds detectors aim to detect objects of interest in point clouds in the format of bounding boxes (BBoxes). Suppose a frame of point cloud data P is a set of point p = [x p , y p , z p , r p ], where (x p , y p , z p ) denotes its 3D location and r p denotes reflective intensity. Thus we can formulate the point cloud detection as: where Det(·) represents the detector; N is the number of detected BBoxes in P; b i denotes i th detected BBox in P, where i = 1, 2, · · · , N ; (x i , y i , z i ) is the Cartesian coordinate of the center of b i , (w i , h i , l i ) is its dimensions, θ i is its heading angle, c i is its classification label, and s i is its prediction confidence score. Point cloud feature representation. Representation for features used in point cloud detection includes 2D-view images, voxels, and raw points. By projecting point clouds into a 2D bird's eye view or front view, 2D-view-based 3D detectors can intuitively fit into a 2D image detection pipeline [34], [35]. However, 2D-view images could lose depth information [2], where the localization accuracy of the detector is affected. To efficiently acquire 3D spatial knowledge in large-scale point clouds, the "voxelization" operation is leveraged to partition unordered points into spatially and evenly distributed voxels [36], [55]. After pooling interior features, those voxels are fed into a sparse 3D convolution backbone [36] for feature abstraction. Given an appropriate voxelization scale, voxel-based representation is computationally efficient, but the quantization loss by voxelization is also inevitable [2]. Different from the above methods, PointNet [56] and PointNet++ [57] directly extract abstract features from raw points, which keeps the integrity of spatial context in point clouds. However, the point-based detectors are not cost-efficient for large-scale data [2]. As a trade-off between the voxel-based and point-based methods, Point-voxel-based representations [40], [41] possess the potential of fusing the high-efficient voxels and accurateabstract points in feature abstraction. Proposal architecture. One-stage detectors [36], [50] directly generate candidate BBoxes from the abstracted features. To improve candidate BBoxes' precision, two-stage detectors [40], [13] refine those BBoxes by region proposal network (RPN) and tailor them into unified size by region of interest (RoI) pooling before predicting output BBoxes. Compared with one-stage detectors, two-stage ones [2] usually present more accurate localization but intuitively, are more computationally timeconsuming.

B. Robustness Enhancement Solutions
Several attempts have been made to enhance the robustness of point cloud detectors. In this paper, we select data augmentation and denoising methods to study their effects on improving point cloud detectors' robustness against common corruptions. Data augmentation [58] is an effective way of increasing the amount of relevant data by slightly modifying existing data or newly creating synthetic data from existing data. Data augmentation on the point cloud [46], [59] provides detectors with a way to be trained with a larger dataset and thus potentially obtain more robust detectors. Different from data augmentation, denoising [51], [52] serves as a pre-process to detect and remove spatial outliers in point clouds, which can reduce the effects of noisy point cloud data.

IV. PHYSICAL-AWARE ROBUSTNESS BENCHMARK FOR POINT CLOUD DETECTION
We propose the first robustness benchmark of point cloud detectors against common corruption patterns. We first introduce different corruption patterns collected for this benchmark and dataset in Section IV-A. Then we propose the evaluation metrics used in our benchmark in Section IV-B. Finally, we introduce the subject-object detection methods and robustness enhancement methods selected for this benchmark in Section IV-C.

A. Physical-aware Corrupted Dataset Construction
After the literature investigation in Section II-A, we summarize 25 corruption patterns in Table II and categorize them into 4 categories based on presentations of common corruptions: weather, noise, density, and transformation. On the other hand, we also divide common corruption patterns into the scenelevel and the object-level. As an initial effort, the dataset covers representative but not all corruptions, and we encourage continuous work with more diverse corruptions considered in the future.
The simulation of corruptions implemented in the paper mainly operates on the spatial locations and the reflection intensity of points in the point cloud. Those point-targeting operations are equivalent to the perturbations of the real-world corruptions on the LiDAR point cloud and have been widely utilized in the simulation-related studies, as in noise-related [24], [23], [18], [11], [46], density-related [18], [11], [46], [60], [13], and transformation-related [18], [11], [29], [27], [60]. Next, We briefly introduce each corruption pattern in the following (refer to Appendix C for detailed implementations and visualizations). Weather corruptions: LiDAR is sensitive to adversarial weather conditions, such as rainy, snowy, and foggy [9]. Dense droplets of liquid or solid water dim the reflection intensity and reduce the signal-to-noise ratio (SNR) of received lights. Floating droplets can also reflect and fool sensors with false alarms. Both effects, in some cases, can significantly affect the detectors. To simulate three weather corruptions: {rain,  snow, fog}, we adopt LiDAR light scattering augmentation (LISA) [12] as a simulator for rain and snow and LiDAR fog stimulation (LFS) [48] as a fog simulator.
To verify the naturalness of weather simulation, we train weather-oriented PointNet-based classifiers with datasets collected in real snowy and foggy weather. Then, we leverage the classification accuracy of those trained classifiers testing on simulated data to measure the similarity of simulated data to real data. As shown in Table III, the testing accuracy 97.13% and 92.60% of trained weather classifiers on simulated snow data and fog data show that the simulated snow data and fog data are highly similar to the real data (refer to Appendix B-A for detailed experiment settings).
We further analyze the similarity of distribution of real and simulated corrupted data. Specifically, we extract the highlevel features from the trained classifier. Then, we utilize T-SNE [61] to reduce the dimensionality of acquired features to 2 and visualize these 2D features. As shown in Figure 1 and 2, the distributions of the real and simulated corruptions are significantly similar. We further quantitatively measure the distance between the feature distribution of clean data, simulated snow/fog, and real snow/fog, as shown in Table  IV. The maximum mean discrepancy (MMD) [62] results reveal that the simulated snow/fog is close to the real snow/fog, respectively, while not close to the clean data.  Regarding rain corruption, we find the effects of rain droplets on point clouds are too subtle to be caught by classifiers, as shown in Figure 3. Alternatively, we visualize simulated and real point clouds and qualitatively verify the high similarity between simulated and real point clouds (see more comparisons in Appendix B-A). Noise corruptions: Noise commonly exists in point cloud signals [53], [51]. Scene-level factors (e.g., strong illumination [22], limited ranging accuracy of sensors, and sensor vibration [24], [25]) could increase the variance of ranging or extend the positioning bias. Floating particles, e.g., dust [26], could cause the background noise in point clouds. Hence, we collect 5 scene-level noise corruptions: {uniform_rad, gaussian_rad, impulse_rad} add uniform, Gaussian, impulse noise on the spherical coordinates of points in point clouds; {upsample} randomly upsamples points nearby original points in point clouds; {background} uniformly randomly samples points within the spatial range of point clouds. Besides scene-level effects, objectrelated factors could cause noise in LiDAR points, e.g., dark color [21] and coarse surface. Thus, we formulate 4 objectlevel corruptions: {uniform, gaussian, impulse} add uniform, Gaussian, impulse noise on the Cartesian coordinates of points of objects; {upsample} upsamples points nearby original points of objects. Density corruptions: The density-related corruptions refer to the corruption patterns that change the global or local density distribution of LiDAR point clouds. For instance, the global static density of points in LiDAR varies due to different amounts of scanning layer (e.g., 32 or 64). Besides, inter-object occlusion and random signal loss [13] could remove points randomly. We hence propose 5 corruptions: {cutout} cuts out the sets of locally gathering points; {local_dec, local_inc} locally decrease or increase the density of points; beam_del, layer_del randomly delete points or layers of points in point clouds. In terms of object-level factors, dark-color cover [21] and transparent materials (e.g., glasses and plastics) of objects can affect the point density of objects. Hence, at the object level, we also propose a set of corruptions: {cutout, local_dec, local_inc}, affecting the point density of objects. Fig. 3: Comparison between real rain and simulated rain (red and yellow boxes contain the false points in the simulated and real rain, respectively; the data of real clean and real rain from Boreas were sampled at the same location; the simulated rain data was augmented on the basis of the real clean data) Transformation corruptions: In the scenario of autonomous driving, shapes of objects within one class could be various (e.g., flat sports cars and round vintage cars [29], bending and walking pedestrians [28]). Those long-tail data could potentially be recognized wrong. Besides, dynamic changes in heading directions and locations of objects [27] could potentially affect the positioning accuracy of detectors. Hence, we formulate 5 corruptions: {translation, rotation} change locations and heading directions of objects to a milder degree, i.e., < 1m and < 10 • ; {shear} [63] and {scale}, as linear deformations, slant and scale points of objects; {FFD} adopts free-form deformation (FFD) [64] to distort the point shape of an object in a nonlinear manner. Dataset selection. As one of the most popular benchmarks in autonomous driving, KITTI [6] contains 7481 training samples covering 8 object classes. Unlike other datasets in Table I, the data in KITTI are mostly collected under clean conditions and also have a relatively simple annotation format, which makes it a good option for conducting comparative experiments. We also encourage the future extension to other real or synthesized datasets. To simulate various levels of severity in the real world, we set 6 severity levels for each corruption (considering "clean" as level 0).

B. Evaluation Metrics
To quantify the robustness performance of detectors, we design the following evaluation metrics from two perspectives: (1) detection accuracy and (2) number of bugs triggered. Overall accuracy. For each test, we use the overall accuracy (OA), by taking the average of APs (average precision) at three difficulty levels (i.e., "Easy", "Moderate", and "Hard"). And we follow the common settings of IoU thresholds {Car: 0.7, Pedestrian: 0.5, Cyclist: 0.5} to search for the true positive detections in AP and recall calculation.
For every corruption, we calculate corruption error (CE) to measure performance degradation according to OA by: where OA m c,s is the overall accuracy of detector m under corruption c of severity level s (exclude "clean", i.e., severity level 0) and clean represent the clean data. For detection m, we can calculate the mean CE (mCE) for each detector by: Detection bug. There are various bugs existing in the pipeline of point cloud detection, such as annotation errors, run-time errors, detection bugs. In this paper, we focus on the bugs in detection results. Specifically, we're interested in false detection, false classification, and missed detection: where * stands for FD, FC, and MD; N * is the number of objects of * ; N det is the number of detected objects.
To measure the increase of BR after being affected by common corruptions, we calculate corruption risk (CR) and the mean CR (mCR) for detector m by where BR m * ,c,s is the BR * of detector m under corruption c of severity level s.

C. Benchmark Subjects
Point cloud detectors. For benchmarking point cloud detection, we select 8 representative detectors: SECOND [36], PointR-CNN [38], PVRCNN [40], BtcDet [13], VoTr-SSD, VoTr-TSD [37], Centerpoint [54], and SE-SSD [65] to cover different kinds of feature representations and proposal architectures. We show the detailed taxonomy in Table VII. Data augmentation and denoising methods. In this paper, we study the effectiveness of data augmentation and denoising    [46]. For denoising, we adopt K-nearest-neighbors-based outlier removing (KNN-OR) [53] to remove the outliers out with 3 times the standard deviation of distance distribution within the cluster of 50 points.

V. EXPERIMENTS AND ANALYSIS A. Experimental Set-ups
For a fair comparison, each detector in Table VII is trained with the clean training set of KITTI, following the training strategy in each paper, and evaluated with corrupted validation sets of KITTI. All detectors are executed based on the opensource codes released on GitHub, as shown in Table XIV in the Appendix A. The configuration files and pre-trained checkpoints can be found in Table XIV. The training and evaluation are all executed on the NVIDIA RTX A6000 GPU with a memory of 48GB. The batch size of each detector is optimized to reach the limit of GPU memory. In the experiments of robustness enhancement, data augmentation is adopted to augment the clean train dataset before training and the denoising directly processes the val data during the testing stage. Note that, since only detection of "Car" is available for all detectors, as shown in Table V, the following evaluation will mainly focus on detected results in the "Car" category. We encourage readers refer to the Appendix A for complete evaluation results, e.g., about "Pedestrian".

B. Effects of Common Corruptions to Point Cloud Detectors
How do different corruptions affect detectors' overall accuracy? As shown in the yellow cell in Table VI, the average mCE AP of 11.01% anticipates a noticeable accuracy drop of detectors against diverse corruption patterns. These results suggest that there is an urgent need of addressing the point cloud detector's robustness issue.
Specifically, {rain, snow} and {shear, FFD} corruptions have the AP loss of more than 20% (last column in Table VI), which presents a serious degradation of detection accuracy. By contrast, some corruption patterns (e.g. scene-level and  object-level upsample, scene-level beam_del, and object-level rotation) show less effects on detectors (CE AP less than 1%). It demonstrates that upsampling noise, sparse beam loss, and slight rotation don't affect detectors' accuracy. Besides, as shown in Table VIII, the recall metric performs similarly to AP, as the serious recall loss of over 20% happens to {rain, snow, shear}. In addition, object-level {cutout, local_dec, FFD} present an unignorable drop of recall within [15%, 20%]. How do corruption severity levels affect detectors' overall accuracy? We find almost all common corruptions have a predictable trend, i.e., each corruption's CE AP increases as its severity level increases (see Table IX for detailed results). The only exception is rain, CE AP of which remain around 26% regardless of the severity level. There are two plausible explanations: (1) noise points reflected by rain droplets are too sparse to affect detection (see Figure 3), and (2) a vast amount of points with zero-value reflection intensity in KITTI are not affected by rain corruptions at 1-5 severity levels, which cause a fixed accuracy drop on point cloud detection.

C. Reacts of Detector Designing to Common Corruptions
How do different representations affect detectors? As shown in Figure 4, voxel-based Centerpoint and BtcDet record the lowest and highest CE AP . For voxel-involving detection (i.e., except for PointRCNN), mCE AP approximately increases as AP increases. It indicates a potential trend that more accurate models trend to become less robust against common corruptions. We also find that, for the most of corruptions (except {shear, FFD, scale}), voxel-based methods are generally more robust against corruption patterns (as shown in Table VI). One plausible explanation is that the spatial quantization of a group of neighbor points by voxelization mitigates the local randomness and the absence of points caused by Noise and Density corruptions. Specifically, for severe corruptions (e.g., shear, FFD in the Transformation), the point-voxelbased method PVRCNN is more robust. The point-based PointRCNN doesn't have the most robust performance against any corruption, suggesting potential limitations. How do different proposal architectures affect detectors? As shown in Figure 4, two-stage detectors perform less robust against common corruptions compared to one-stage detectors, showing a lower mCE AP . One possible cause is that corrupted data could affect the proposal generation of stage 1 (for two-stage detectors and one-stage ones), and the low-quality proposals significantly affect the BBox regression of stage 2 (only for two-stage detectors).
As shown in Table X, two-stage detectors present more accurate detection under the scene-level {cutout, local_dec, beam_del, layer_del} and object-level {cutout, shear, scale}, displaying a lower average CE AP , while one-stage detectors present more accurate under the rest of common corruptions. In summary, one-stage detectors perform more robust against corruptions of scene-level Noise and object-level Noise and Density, while two-stage detectors are mainly more robust

D. Detection Bugs in Detectors under Common Corruptions
How do different corrupted inputs trigger bugs in detectors? We find that the rate of false classification (FC) against common corruption patterns is relatively small, where the largest CR F C is only 0.97% (refer to  milder noise points offer a better knowledge of the shape of some objects to detectors, but positioning on those objects is not accurate since the rate of false detection increases (more details in Table XVI and XVII in Appendix A-C). Also, we find that, as shown in Figure 5, compared to clean observations, TD rates under corrupted observations are always lower at any distance of objects to LiDAR. How do corrupted inputs trigger bugs in different detectors? In general, as shown in Table XII, most of the detectors perform relatively stable in terms of false classification rates and missed detection rates against common corruptions. In contrast, affected by corruptions, all detectors have increasing false detection rates (refer to Table XII), revealing a serious bias in BBox localization of point cloud detection. Among all detectors, BtcDet and SE-SSD records a serious FD increase   Table XIII, the average CE AP with PA-DA slightly decreased to 10.85% compared to the average CE AP without PA-DA, which still poses serious robustness issues for point cloud detectors.
Regarding denoising strategy, the average CE AP after adopting KNN-RO increases to 13.60% without PA-DA and 13.38% with PA-DA (refer to Table XIII). These results indicate that KNN-RO might not be capable of enhancing point cloud detectors' robustness against most of the corruptions in Car detection. However, we find that KNN-RO slightly improves the robustness of Pedestrian detection by decreasing the CE AP by 0.14% without PA-DA and 1.19% with PA-DA (Table XVIII in Appendix A-D). How do PA-DA and KNN-based outlier-removing affect different detectors' robustness against corruptions? Except for PVRCNN, SE-SSD, and Centerpoint, all the other detectors perform more robust against corruption patterns after adopting PA-DA (refer to Figure 6). Moreover, PointRCNN and VoTr-SSD increase their AP by 1.16% and 2.46% after adopting PA-DA, respectively.
According to Figure 6, KNN-RO degrade AP metric for all detectors, presenting no improvement on the robustness of any detector on Car detection. However, adopting KNN-RO slightly improves the AP by 0.14% without PA-DA and 1.19% with PA-DA on Pedestrian detection, respectively (refer to Table XIX in Appendix A-D). It illustrates that compared with effects on Car objects, KNN-RO are more effective in

VI. CONCLUSION
In this paper, we propose the first physical-aware robustness benchmark of point cloud detection against common corruption patterns, which contains a total of 1,122,150 examples covering 25 common corruption types and 6 severity levels. Based on the benchmark, we conduct extensive empirical studies on 8 detectors covering 6 different detection frameworks and reveal the vulnerabilities of the detectors. Moreover, we further study the effectiveness of existing data augmentation and denoising methods and find them limited, calling for more research on robustness enhancement. We hope this benchmark and empirical study results can guide future research toward building more robust and reliable point cloud detectors.

APPENDIX A DETAILS OF EMPIRICAL STUDY
Due to the page limit of supplementary materials, we present some tables and figures on our supplementary website https://sites.google.com/ualberta.ca/robustness1pc2detector/. We also encourage readers to refer to this supplementary website for additional details.

A. Effects of Common Corruptions to Point Cloud Detectors
As shown in Table S1 in the supplementary, the average mCE AP of 8.18% still anticipates a noticeable accuracy drop of Pedestrian detectors against diverse corruption patterns. Specifically, scene-level {uniform_rad, gaussian_rad, local_dec} and object-level {cutout} corruptions have the AP loss of more than 20%, which presents a serious degradation of detection accuracy. By contrast, some corruption patterns (e.g. scene-level {background, beam_del}, object-level {upsample, rotation, translation}) show less effects on detectors (absolute value of CE AP less than 1.25%), demonstrating that background and locally upsampling noise, sparse beam loss, and slight rotation and translation don't affect Pedestrian detectors' accuracy. Surprisingly, compared to Car detection, Pedestrian detection are much less affected by rain and snow, presented by the average CE AP of 2.42% and 2.58%. After the investigation of point clouds, we found it is because the proportion (58.94%) of points with zero-value reflection intensity on Car objects is much higher than that (10.01%) of pedestrians, and those points with zero-value reflection intensity are easy to be blocked by dense rain and snow droplets. Table S2 in the supplementary shows the CE AP under different severity levels of corruptions on Pedestrian detection. According to Table S2, albeit with some minor exceptions, CE AP of each corruption increases as the severity level increases, which especially rigorously applies to those relatively severe corruptions with the average CE AP of more than 5%. Figure S1 in the supplementary depicts the relationship between of AP and mCE AP of Pedestrian detectors. As shown in Figure S1, similar to Car detection, the mCE AP of Pedestrian detection increases as its AP increases.

C. Detection Bugs in Detectors under Common Corruptions
More details of the increase of bug rates (i.e., CR) of different detectors under different corruptions are recorded in Table XV, Table XVI, and Table XVII. According to Figure 5, the TD rate of Car detection under clean and corrupted (at the severity level of 3) observations w.r.t. the distance range of [0, 30]m remains approximately still. However, beyond this distance range (i.e., distance > 30m), the TD rate decreases as the distance increases. On the other hand, compared to the clean observation, the TD rates under corrupted observations are always lower at any distance. More details of each corruption are shown in Figure S2 and S3 in the supplementary.

D. Robustness Enhancement by Data Augmentation and Denoising
More details of CE AP of different detectors on Car detection with PA-DA (data augmentation) and/or KNN-RO (denoising)  under different corruptions are shown in Table S3, Table S4, and Table S5 in the supplementary. Table XVIII gives the details of average AP loss (i.e., CE AP ) given different corruptions on Pedestrian detection with PA-DA (data augmentation) and/or KNN-RO (denoising). According to Table XVIII, Table XIX shows CE AP of different detectors on Pedestrian detection with PA-DA (data augmentation) and/or KNN-RO (denoising). According to Table XIX, with Table III).    [56], which fits into the global effects of weather corruptions on LiDAR scanning. Analysis: According to the snow classification in III, the validation accuracy of 99.11% illustrates the snow classifier's precise snow recognition on point clouds collected in the real world. Hence, the classifier's testing accuracy of 97.13% on simulated data demonstrates the high naturalness of the snow simulation. Likewise, even though the fog classifier's validation accuracy is reduced by the small data size, the testing accuracy of 92.60% on simulated data still presents a high similarity of data corrupted by simulated fog to data affected by real fog.
2) Rain Validation: In Figure 3, data of real clean and real rain from Boreas were sampled at the same location but at different time-stamps; the data of simulated rain was augmented on the basis of the data of real clean. According to Figure 3, compared with the data of real clean, the point cloud under simulated rain has sparse "false points" (as in red boxes) surrounding the LiDAR sensor nearby and wipes out some points on the road. Both are similar to the effects of real-world rain, shown by the yellow boxes and missing points on the road. More comparisons between real rainy data and simulated rainy data are shown in Figure S4 in the supplementary.  Beam_del. For beam_del, we randomly delete a certain portion of points in the point cloud. The portion is set to {0, N/100, N/30, N/10, N/5, N/3} for 6 severity levels. Figure S17 displays the point cloud example under beam_del. Layer_del. First, we convert the coordinates of points from the Cartesian system into the spherical system and obtain the range of the polar angle θ of all points in the point cloud. Then, based on the layer number of LiDAR scanning (e.g., 64 for KITTI [6]), we divide the range of θ into 32 or 64 bins. Finally, we randomly select a certain number of θ bins and delete the corresponding points. For KITTI, the number of deleted bins is set to {0, 3, 7, 11, 15, 19}. Figure S18  Upsample_obj. We upsampled points (with the spatial bias within [-0.05, 0.05]) nearby a certain portion of points of annotated objects. The portion is set to {0, N/5, N/4, N/3, N/2, N }, where N represents the number of original points of objects. Figure S22 displays the point cloud example under upsample_obj. Cutout_obj. For cutout_obj, we erase the distance-related neighborhoods of selected points from every annotated object in the point cloud. For the individual object, the number of selected points and the number of neighbor points are set to {(0, 0), (1,20), (2,20), (3,20), (4,20), (5,20)}. Figure S23 displays the point cloud example under cutout_obj. Local_dec_obj and Local_inc_obj. For local_dec_obj, we randomly select a certain number of points of every annotated object as centers, and delete the 75% points of the spatial neighborhood of every center. For local_inc_obj, within the neighborhood of every centers on objects, we utilize linearly fitting to upsample points as many as the neighbor points. The number of selected points and the number of neighbor points are set to {(0, 0), (1,30), (2,30), (3,30), (4,30), (5,30) The scaling parameter randomly selected among xs, ys, zs are set to 1+/−{0, 0.04, 0.08, 0.12, 0.16, 0.20}. Note that for scaling on Z-axis, we correspondingly move the object to the ground. Also, the ground-truth labels of objects (specifically, dimensions and locations of BBoxes) are modified accordingly. Figure S28 displays the point cloud example under scale. Rotation and Translation. We rotate or translate annotated objects to a milder degree. Specifically, objects are 1) moved forward or backward on the X and Y-axis at a distance sampled on the uniform distribution U distance , and are 2)rotated in a clockwise or anticlockwise direction to a degree sampled on the uniform distribution U degree . The lower and upper boundaries of U degree are set to {(0, 0), (0, 2), (3,4), (5, 6), (7,8), (9,10)