{'fraction_non_alphanumeric': 0.021763351407561464, 'fraction_numerical': 0.0084305132585605, 'mean_word_length': 4.937596471885336, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pattern-Aware Data Augmentation for LiDAR 3D Object Detection

Autonomous driving datasets are often skewed and in particular, lack training data for objects at farther distances from the ego vehicle. The imbalance of data causes a performance degradation as the distance of the detected objects increases. In this paper, we propose pattern-aware ground truth sampling, a data augmentation technique that downsamples an object's point cloud based on the LiDAR's characteristics. Specifically, we mimic the natural diverging point pattern variation that occurs for objects at depth to simulate samples at farther distances. Thus, the network has more diverse training examples and can generalize to detecting farther objects more effectively. We evaluate against existing data augmentation techniques that use point removal or perturbation methods and find that our method outperforms all of them. Additionally, we propose using equal element AP bins to evaluate the performance of 3D object detectors across distance. We improve the performance of PV-RCNN on the car class by more than 0.7 percent on the KITTI validation split at distances greater than 25 m.


I. INTRODUCTION
3D object detection has emerged as a prominent challenge on the way to advanced autonomy systems, especially for the application of autonomous driving. Deep learning techniques have dramatically advanced progress in this domain, however, much remains to be done. One active area of research lies in advancing the capabilities of 3D object detection at long range. Rather than purchasing expensive sensors for more accurate 3D object detection, algorithms can be specifically designed for robust detections in the far range setting as a cheaper alternative. Along with autonomous driving, accurate long range object detection has many other potential applications as well, from intersection traffic monitoring, aerial vehicle security to long-range inspection of infrastructure, for example.
LiDAR is an integral sensor for autonomous driving and many of the state-of-the-art (SOTA) methods rely entirely on this one sensor [5], [8], [9], [13], [14]. LiDARs use laser light to measure distances, creating a 3D point cloud of the scanned environment with high depth accuracy. The captured point cloud can be processed through various methods for accurate 3D object detection.
However, there is still room for improvement in current LiDAR 3D object detectors, particularly at farther distances. The performance of object detectors tends to degrade as the distance from the LIDAR increases, as shown in Figure  6. The problem with detecting objects at far range is twofold. The first reason stems from the LiDAR itself. LiDARs such as the Velodyne HDL-64E use a series of vertically positioned lasers, creating scan lines across the environment. However, as the distance from the sensor increases, the distance between points increases. This is inherent to the angular resolution of the LiDAR. Thus, the point cloud density on detected objects decreases as we increase the distance from the sensor. The reduction of point cloud data on the detected objects makes it difficult for the algorithm to predict accurate bounding boxes and the performance therefore suffers as a function of range to an object. The second reason is the lack of training data. For  We propose a solution to resolve the imbalance in training data through a novel pattern-aware ground truth augmentation. By artificially downsampling the point cloud data based on the LiDAR characteristics as shown in Figure 1, we are able to obtain more training examples at farther distances and subsequently increase the performance of detected objects at far ranges.
The benefit of different architectures or data augmentations is usually determined by the average precision (AP) score on the associated dataset. Indeed, AP offers a simple metric to quickly evaluate different algorithms against each other to show the relative value of new contributions within the field. However, the caveat is that it is hard to distinguish the benefit of different methods from a single number. For example, an object detection algorithm can focus on close range examples to boost its overall performance on the benchmark, but is ultimately a poor object detector given its poor performance at farther ranges.
A better method, similar to the Waymo Open Dataset [15] evaluation metric, is to use distance based AP evaluations. However, naively evaluating samples in equal bin widths has a few issues. As mentioned previously, the data is skewed in KITTI across distance. If the dataset is naively partitioned into equally sized bins (such as 10-20 m, 20-30 m, etc.), there will be an unequal number of samples in each bin. The resulting AP score in each bin therefore is also dependent on the number of samples within said bin. This could result in a poor representation of the object detector's performance in bins with a small number of samples. Rather, we propose to use bins with an equal number of elements and varying size. Thus, the evaluation will not only capture the data evenly, but will also be flexible to any dataset distribution.
To summarize, our contributions are as follows: • We propose a novel method for creating farther distance objects through pattern-aware ground truth sampling • We increase the performance of PV-RCNN by more than 0.7% at distances greater than 25 m for the car class on the KITTI validation split • We demonstrate the value of using equal element AP bins across distance to capture more insight into the performance of object detectors • We evaluate our approach against other methods that use point removal or perturbation techniques and find that our method outperforms all of them.

A. Data Augmentations in LiDAR 3D Object Detection
Following Hahner et al.'s [7] categorization of different data augmentations, a distinction is made between global and local augmentations. Global augmentations are when the entire point cloud is transformed by the specified augmentation. An example of this would be scaling the entire point cloud by a specified factor. A local augmentation only operates on the points within a ground truth box, such as rotating a bounding box and the associated points. Therefore, local augmentations can take into consideration the local characteristics of the ground truth box, such as the class or the number of points within the box, whereas global augmentations operate on the point cloud as a whole.
VoxelNet [20] uses affine transformations for augmenting point clouds in autonomous vehicle datasets and these augmentations are widely used in SOTA models. These include scaling, rotation, and translation of the point clouds. A transformation factor is typically sampled from a random distribution to slightly perturb the point cloud. PIXOR [18] uses another affine transformation in the form of randomly flipping the point cloud on the vertical axis, similar to the vertical flip augmentation in images. Affine transformations can be performed as global or local point cloud augmentations, but they do not take into consideration the characteristics of the LiDAR when perturbing the points.
SECOND [17] introduces ground truth augmentation. First, a database that contains all the 3D ground truth boxes is created. During training, the database is sampled to insert additional ground truth samples into the current point cloud sample. If any new ground truth boxes overlap with existing object boxes, they are not included in the point cloud. Ground truth sampling does not alleviate the imbalance in training data because the sampled database has the exact same distribution of the original point cloud data across distances.
Recently, new methods for augmenting point clouds have been proposed. PointMixup [1] interpolates data between examples through an optimal assignment of the path function between two point clouds. Choi et al. [3] introduces PA-AUG, which divides objects into partitions and stochastically applies different augmentations to the segmented portions. Again, these methods treat the point clouds strictly as 3D points in space without considering the sampling pattern of the sensor.
There has also been work on automating the data augmentation parameters throughout training. PointAugment [10] uses an adversarial learning strategy where the augmentation network and the classifier network are jointly trained to create a classifier that is robust to intra-class data variations. Progressive Population Based Augmentation (PPBA) [2] uses an evolutionary search algorithm to create a data augmentation scheduling policy.
PPBA also uses additional local data augmentations to increase examples of occlusions and low point density: frustum dropout, frustum noise, and random drop. Both frustum dropout and frustum noise first define a frustum for each ground truth box where the direction of the frustum points from the LiDAR sensor origin to the bounding box. Frustum dropout randomly drops points within the intersection between the ground truth box and the defined frustum with a fixed probability. Frustum noise works similarly but instead perturbs the intersected points with noise sampled from a Gaussian distribution. Random drop simply removes points within a ground truth box based on a fixed probability. These augmentations are similar to pattern-aware ground truth sampling as they attempt to remove or perturb points in the point cloud but they do not consider the sampling pattern of the LiDAR sensor.

B. Sensor Simulation
There are also methods that attempt to simulate LiDAR data. Tallavajhula et al. [16] uses real data to create vegeta-tion and off-road LiDAR simulations. LiDARSim [12] uses raycasting over 3D scene data and a deep neural network to produce realistic LiDAR point clouds. Other techniques include using CAD models to simulate LiDAR data such as Fang et al. [4] who combine real world background models and simulated data for improved 3D object detection.

III. METHOD
In this work, pattern-aware ground truth sampling is proposed and evaluated against both common augmentation techniques introduced in SECOND [17], as well as additional augmentations outlined in PPBA [2]. As mentioned in Section II-A, the data augmentations used in PPBA (frustum dropout, frustum noise, and random drop) serve a similar purpose as pattern-aware ground truth sampling by removing or perturbing points. Thus, these augmentation methods serve as an additional baseline for comparison.
Inspired by PseudoLiDAR++'s downsampling method to simulate cheaper 4 and 8 beam LiDARs [19], pattern-aware ground truth sampling downsamples point cloud data based on the LiDAR's characteristics. It builds on top of ground truth sampling [17] by additionally downsampling ground truth boxes and moving them farther in the scene, simulating farther objects. Pattern-aware ground truth sampling is a local augmentation, operating on each ground truth box.
Pattern-aware ground truth sampling considers the characteristics of the LiDAR sensor itself. As objects move farther away from the LiDAR, each laser point diverges from its neighbours. All lasers are emitted at slightly different angles, allowing for a large coverage area. As the distance from the sensor increases, the scan lines of the LiDAR diverge. For example, the Velodyne HDL-64E has a vertical resolution of 0.4 • and a horizontal resolution of 0.08 • to 0.35 • . An object detected at 10 m has a distance of roughly 7 cm between each horizontal scan line whereas an object detected at 40 m has a distance of 28 cm between each scan line. Thus, objects at farther distances will have far fewer scan lines (and points) than closer objects due to the divergence of the lasers from the LiDAR. from the ego vehicle. A point cloud is defined as a set of 3D points {p i |i = 1, . . . , n} where each point p i has spatial coordinates of (x i , y i , z i ). The point cloud can also be represented in its spherical coordinates (r i , θ i , φ i ) as shown in Figure 4. Thus, d i and r i are defined as: and the azimuthal angle θ i and the polar angle φ i are calculated as follows: Once the point cloud is represented in spherical coordinates, the points can be partitioned across both angular dimensions. Let W be the number of azimuthal angle divisions and H be the number of polar angle divisions. Figure  5 shows a bird's-eye-view of the angular partitioning of the point cloud across the azimuthal angle θ. Since the point cloud is partitioned into evenly spaced angular divisions, it can be sampled accordingly to simulate LiDAR data captured at different distances. For example, a ground truth bounding box's distance from the LiDAR can be doubled by sampling every other slice, effectively downsampling the point cloud based on the divergence of the LiDAR points. The pattern-aware downsampling method realistically simulates farther objects as shown in Figure 1. Figure 2 shows an example of pattern-aware ground truth sampling on a point cloud during training. If the original ground truth sampling is used, objects are often clustered at closer distances. By using pattern-aware ground truth sampling, the locations of objects are more varied, allowing the network to train on a diverse set of object distances.
During training, a portion of the regular ground truth samples are moved further into the scene using the stated pattern-aware ground truth sampling algorithm with a fixed probability. This ensures that there is a more even spread of data examples throughout varying distances, allowing the network to learn from data samples at all depths.

IV. EVALUATION
Evaluating objects at different distances is not new as it is a benchmark used for evaluating 3D object detectors on the Waymo Open Dataset [15]. The distance analysis is valuable as it gives an understanding of the performance degradation of the object detection method at detecting farther objects. However, existing datasets such as KITTI do not have benchmarks designed specifically for evaluating performance across distance. Naively dividing the data into equally sized bins may cause evaluation issues. Each bin will have a different number of samples, producing unrepresentative evaluations on bins with a small number of samples. Thus, it is important to use equal element bins to ensure a constant number of samples for a meaningful AP in each bin.
Evaluating using equal element AP bins is simple: first a histogram of the data distribution within the dataset is created with a chosen number of bins where each bin contains the same number of samples. For example, if there are 10 histogram bins, each bin contains a 10th of the dataset, varying the bin widths according to the spread of the data. The height of the histogram is normalized such that each bin has the same area. Thus, wide bins will have a lower height, indicating a spread in the data within that region of the histogram, while narrow bins will be taller, indicating a high concentration of data in that bin interval. Example histograms of the KITTI dataset distribution are shown in Figure 3. The resulting bin widths from the histogram group samples together based on distance and said groups are evaluated using AP. Providing additional bins for evaluation shows general trends in the data that are obscured by the coarse overall AP values common in standard benchmarks, while being flexible to the distribution of the dataset.

V. EXPERIMENTS
All models are trained and evaluated on the KITTI dataset. Training is primarily done using PV-RCNN [13]. Following PV-RCNN, we use the same image split of 3712 training Since the LiDAR used in the KITTI dataset is the Velodyne HDL-64E, 512 azimuth and 64 polar channels are used for the number of angular divisions and downsampled to 32 and 256 channels respectively to double the distance of the original objects' location. The minimum number of points after downsampling the objects through patternaware ground truth sampling is 5, 200, and 200 for the car, pedestrian, and cyclist class, respectively. The range of downsampled objects is limited to 20-70 m from the LiDAR. These point and range limits are used to ensure that objects are not downsampled to too few points. Patternaware ground truth sampling is performed on the samples with a probability of 40%.
Alongside the standard KITTI AP benchmark evaluations, the data augmentation configurations are tested using equal element AP bins to evaluate performance across distance. The bin widths are calculated by splitting the validation ground truth samples into 10 equal element bins. AP is calculated using the predicted and ground truth boxes that fall within the specified bin width. Unlike the standard KITTI benchmark, the distance evaluation also includes ground truth examples outside the KITTI hard difficulty [6] (Minimum bounding box height: 25 pixels, maximum occlusion level: difficult to see, maximum truncation: 50%). This is done to evaluate the object detection performance at far distances which are normally excluded in the standard KITTI difficulty levels. Table I shows the 3D AP results calculated using 40 recall positions for each augmentation. Using pattern-aware ground truth sampling, PV-RCNN obtains a slight overall AP improvement on the KITTI validation split. A good mix of the original and downsampled ground truth augmentations is necessary for improving performance of the object detector.

A. Overall AP
In fact, a 40% probability of pattern-aware downsampling the ground truth augmentations provides a good balance of data examples at various distances. A higher percentage results in a smaller increase in performance as the data overrepresents training examples at far range.
Pattern-aware ground truth sampling is also used on the cyclist and pedestrian classes but it is necessary to set the minumum number of points after downsampling to 200 points. A lower number results in high variation in performance on the two classes. One reason is because of the extreme skewness of data. Figure 3 (b) and (c) show the imbalance in pedestrian and cyclist classes is extreme compared to the car class. Thus, the distribution of data for these classes can drastically change when using pattern-aware ground truth sampling, causing fluctuations in resulting performance. Another reason is because the pedestrian and cyclist classes are less structured than the car class. Unlike downsampling a rigid object, pedestrian and cyclist point clouds can have various spatial configurations, which produces variable results from pattern-aware ground truth sampling if downsampled to a low number of points.
The resulting performance for the PPBA augmentations are mixed. Frustum dropout provides slight increases in the easy and moderate cyclist class but suffers degradations in the pedestrian class. Both frustum noise and random drop suffer degradations in the cyclist and pedestrian class. All PPBA methods perform slightly worse on the car class overall with slight increases in the hard category.
Performance of the PPBA augmentations is improved by coupling frustum dropout and noise together. There are slight increases in most of the categories except in the easy pedestrian class but even the combined augmentations do not perform as well as pattern-aware ground truth sampling.

B. AP Across Distance
The performance gain from pattern-aware ground truth sampling is more evident when the model's performance on 3D AP is plotted using equal element AP bins across distance as shown in Figure 6 and Table II. There is a general increase in AP across distance. Within the 25-41 m and 50-85 m range, there is an increase in of more than 0.7% with the largest increase of 1.49% within the Table II  PV-RCNN 3D AP PERFORMANCE ACROSS DISTANCE ON THE KITTI VALIDATION SPLIT FOR THE CAR CLASS. SAMPLES ARE BINNED AND  EVALUATED ACROSS DISTANCE FROM THE LIDAR. HIGHEST AND SECOND HIGHEST VALUES ARE BOLDED AND UNDERLINED  30-35 m range. Pattern-aware ground truth sampling adds more diverse object locations, increasing performance across distance. The caveat, of course, is the degradation in the 21-25 m and 41-50 m range. This is attributed to the shift in the training data distribution compare to the validation data distribution which is further explained in Section VI-C. Another benefit from pattern-aware ground truth sampling is the increase in AP at closer distances, especially in the 0-9 m range. Within this range, many objects are truncated due to the frustum segmented point cloud. Even though pattern-aware ground truth sampling is specifically targeted towards increasing performance at far range, only objects between the 20-70 m range are downsampled. By reducing the training imbalance across distance, there is a larger percentage of training data at close distances and therefore the trained 3D object detector performs better at closer range. Without the analysis of equal element AP bins, these two observations would not have been easy to discern from the coarse standard KITTI difficulties. Using equal element AP bins provides insightful breakdowns of how a 3D object detector is performing.
The PPBA augmentations overall have some improvements across distance, specifically frustum noise and the combination of frustum dropout and frustum noise. However, pattern-aware ground truth sampling performs substantially better overall. . The data distribution of the car class (a) before and (b) after using pattern-aware ground truth sampling on the KITTI dataset. 5000 ground truth objects are sampled and the distribution of data is plotted. Patternaware ground truth sampling shifts the dataset towards farther objects, creating a less skewed distribution of data.

C. Data Imbalance
To understand the implications of pattern-aware ground truth sampling, its effects on the distribution of the car class data are explored. 5000 data points are randomly sampled from the ground truth database with a fixed seed. Figure 7 shows the balance of histogram distributions of the car class before and after pattern-aware ground truth sampling across distance from the LiDAR sensor. As expected, the first distribution is similar to Figure 3 (a) as the regular ground truth sampling uses the same ground truth boxes as the original training set. By adding pattern-aware ground truth sampling, the dataset shifts to a less skewed distribution, alleviating the inherent imbalance in data and providing the network more training examples at farther distances. Ideally, the resulting distribution across distance should be uniform but doing so currently degrades overall performance. Future improvements to pattern-aware ground truth sampling aim to focus on creating a uniform distribution of training examples across distance while retaining SOTA performance.
Even though the distribution is less skewed after patternaware ground truth sampling, there is a performance degradation in the 21-25 m and 41-50 m bins. As shown in Figure  7, the highest concentration of each distribution is exactly at the 21-25 m and 41-50 m range respectively. Since the models are trained with a distribution closer to Figure 7 (b), the trained models expect less objects in the 20 m range and more objects in the 40 m range, resulting in a performance degradation when evaluating on the validation split which is closer to Figure 7 (a)'s distribution. Nonetheless, the poorer performance in these specific bins are outweighed by the overall performance increase of pattern-aware ground truth sampling across distance.
D. Generalizability Table III shows the performance of the augmentations on SECOND [17], PointPillars [9], and PointRCNN [14] on the KITTI car validation split. Pattern-aware ground truth sampling provides a general increase to SECOND and Point-Pillars, but suffers an overall degradation on PointRCNN. A reason for the variable results in performance may be the differences in architecture. Both SECOND and PointPillars use an established grid of anchor boxes for bounding box predictions, allowing them to encode the pattern-aware ground truth samples in a consistent approach. In comparison, PointRCNN uses a PointNet++ backbone for bounding box proposals and may not benefit from pattern-aware ground truth sampling. The use of farthest point sampling is less structured and may not capture the downsampled ground truth objects well. A possible improvement to pattern-aware ground truth sampling for PointRCNN would be to perform pattern-aware ground truth sampling after the initial farthest point sampling points have been chosen to ensure consistent point sampling.
PointPillars has a tradeoff between the pedestrian and cyclist class performance. The benefit of pattern-aware ground truth sampling does outweigh the drawbacks as there is an overall increase of 2% in the pedestrian class while there is only a small degradation in the cyclist class. Still, patternaware ground truth sampling may need slight tuning to ensure optimal performance. In the case of PointPillars, the cyclist class' minimum number of points for pattern-aware ground truth sampling may need to be increased.

VII. CONCLUSION AND FUTURE WORK
This work presents a method for handling the imbalance of LiDAR data in autonomous driving datasets. Our method utilizes LiDAR characteristics by simulating the diverging lasers to create a more diverse set of training examples across distance. By doing so, 3D object detection methods are able to perform better at detecting objects at farther distances. Compared to other augmentation methods that remove points, pattern-aware ground truth sampling performs better overall. Pattern-aware ground truth sampling is also trained on other LiDAR 3D object detectors to show its applicability to different architectures, providing performance improvements in grid based LiDAR 3D object detectors. Equal element AP bins give key insights on the impact of pattern-aware ground truth augmentations. Although only small increases in performance are shown on the overall 3D AP, the benefit of the method is found at farther distances, particularly at distances greater than 25 m.
Pattern-aware ground truth sampling does not consider reprojecting the removed points when downsampling objects. Realistically, if a car was detected at a farther distance, the removed laser points would hit other objects in the scene. This is true for all point removal methods as they simply remove points without reprojecting them back into the environment. Additionally, the added points from patternaware ground truth sampling are not entirely consistent with the LiDAR scanning pattern. For example, if an augmented object is placed behind another object, the augmented object should also be occluded. A future improvement to patternaware ground truth augmentation would be to add augmented objects that still ensure a realistic LiDAR scan.