{'fraction_non_alphanumeric': 0.03019857624578494, 'fraction_numerical': 0.006294492319220682, 'mean_word_length': 4.4719147191471915, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lidar-based Object Classification with Explicit Occlusion Modeling

LIDAR is one of the most important sensors for Unmanned Ground Vehicles (UGV). Object detection and classification based on lidar point cloud is a key technology for UGV. In object detection and classification, the mutual occlusion between neighboring objects is an important factor affecting the accuracy. In this paper, we consider occlusion as an intrinsic property of the point cloud data. We propose a novel approach that explicitly model the occlusion. The occlusion property is then taken into account in the subsequent classification step. We perform experiments on the KITTI dataset. Experimental results indicate that by utilizing the occlusion property that we modeled, the classifier obtains much better performance.


Introduction
LIDAR is one of the most popular sensors for unmanned vehicle due to its highly precise range measurements. Object detection and classification based on lidar point cloud is an extremely important technology for unmanned vehicle. However, the sparseness of the lidar point cloud and the mutual occlusion between neighboring objects poses significant challenges for object detection and classification algorithm. Fig.1 is a typical traffic scene. We could observe a lot of occlusions occurred in this figure.
Ideally, the lidar point cloud corresponding to an object should be relatively complete and fully reflect the spatial distribution characteristics of objects. However, due to the mutual occlusion of neighboring objects, the object point cloud is usually incomplete which may result in the wrong classification of the object.
An illustrative example is shown in Fig.2. In the training phase, many positive samples, including sample A and B as shown in the top row of Fig.2, are fed into the classifier. It is seen that sample B is occluded by another obstacle, making its point cloud incomplete. The classifier is then trained to adapt to this intraclass variation. In the testing phase, the classifier encounters two samples, C and D. Among them, sample D is a true positive while C is composed of two small objects, E and F. The classifier will encounter difficulties in distinguishing C from D, and it is very likely to classify C as a false positive or classify D as a false negative.
In this paper, we consider occlusion as an intrinsic property of the point cloud data. The occlusion area could be accurately computed by considering the relative position between the LIDAR itself and each detected LIDAR point using ray-casting technique [1]. Therefore, we add a pre-processing step to add the occlusion property to the point cloud before any further processing. As shown in the bottom row of Fig.2, the occluded area is colored in yellow. With the help of the occlusion area, the classifier can now easily distinguish object C from D. Therefore, both the false positive rate and false negative rate might be reduced.
We test our approach on the KITTI dataset. We choose to use PointNet [2] as the basic classifier. We modified PointNet to enable it to utilize the occlusion property. Experimental results show that our method obtains a significant improvement compared to the original PointNet, both in the overall classification accuracy and per-class classification accuracy.

Related Work
There has been a large literature on object detection approaches based on point cloud. Petrovskaya et al. proposed an object detection algorithm based on object geometry and motion model [3,4,5] , and used Bayesian filters to estimate the model parameters. Himmelsbach et al. extracted the geometric features of point cloud using the point feature histogram [6,7] , and then used SVM to classify the object. Built on the work of [3,4,5] , Wojke et al. [8] proposed an object detection algorithm based on the combination of line features and angular features. Cheng et al. proposed to use histogram features for object detection and recognition [9] . In the training phase of the traditional approach, many positive samples, including sample A and B as shown in the top row, are fed into the classifier. Sample B is occluded by another obstacle, making its point cloud incomplete. In the testing phase, the classifier encounters two samples, C and D. Sample C is likely to be classified as false positive while sample D is likely to be classified as a false negative. In our approach, we add a pre-processing step that computes the occlusion property of the point cloud before the classification module. As shown in the bottom row, the occluded area is colored in yellow. With the help of the occlusion area, the classifier can now easily distinguish object C from D, thus both the false positive rate and false negative rate might be reduced.
Recently, deep learning based approaches have become popular due to its outstanding performance. MV3D [10] firstly projects point cloud onto the bird's eye view and then trains a region proposal network (RPN) for generating 3D bounding box proposals. However, MV3D does not perform well in detecting small objects such as pedestrians and cyclists. VoxelNet [11] is an end-to-end object detection framework. It divides the point cloud into equally spaced threedimensional voxels and then transforms the points in each voxel into a uniform feature representation through the newly introduced Voxel Feature Coding (VFE) layer. The point cloud is then encoded as a volume representation to perform the detection and classification. Different from those previous approaches that rely on a mid-level representation, such as the image grids or the 3D voxels, Qi et al. proposed a new type of network called PointNet [2] that works directly on the original point cloud. PointNet is a unified framework that can be applied to object classification, part segmentation and scene semantic parsing. It obtains competitive results on several 3D object classification benchmarks.
For occlusion handling, there have been several works [12,13,14,15] trying to directly predict the occlusion mask. However, most of these works are imagebased approaches. There has been little work on lidar-based approaches that directly models occlusion and utilize the occlusion property to aid the classification tasks.

Point Cloud Definition
A point cloud is represented as a set of three dimensional points {P i |i = 1, ..., n}, where each point P i is a vector of (x, y, z).
We define the object point cloud data within the object bounding box as the object point cloud P raw = P 1 raw , P 2 raw , ..., P n raw . The point cloud outside the object bounding box is defined as the obstacle point cloud P ob = P 1 ob , P 2 ob , ..., P m ob . The obstacle point cloud will block the lidar ray from passing through it, thus resulting in an incomplete object point cloud. In Fig.3, we can see that the object point cloud is divided into two parts. The occlusion area generated by the point cloud using the ray-casting technique is defined as the occluded point cloud P oc = P 1 oc , P 2 oc , ..., P k oc , and is colored in yellow and pink respectively.

Occlusion Area Modeling
For each point P i ob = x i ob , y i ob , z i ob , i = 1, 2, 3, ..., m of the obstacle point cloud and each point P j raw = x j raw , y j raw , z j raw , j = 1, 2, 3, ..., n of the raw object point cloud, we use the ray-casting technique to model the occlusion. We define the position of the LIDAR as the origin O. For each point P i ob and P j raw , we add occluded points P l oc along the direction of O to P i ob or P j raw at a fixed step. The occluded points are added until their height is below the ground plane. The ground plane is estimated by using a block recursive Gaussian process regression algorithm [16].
For each point P i ob of the obstacle point cloud: For each point P j raw of the object point cloud: where k 1 , k 2 are positive integer, s is the step size ( in our experiment we set s = 0.3m). The function L(x) represents the distance from the point P to the origin O.
To distinguish the added occluded point cloud from the original point cloud, we add a new dimension named 'occluded' to the original point cloud data, expanding the point cloud dimension from three dimensional (x, y, z) to four dimensional (x, y, z, o). We set the occlusion property of original point clouds to 0 and set the occlusion property of newly added occlusion points to 1. We use our approach to add the occluded point cloud for both the object and the obstacle point cloud.
We show the comparison of the object point cloud with and without the occluded points in Fig.4. The first row and the third row are the raw object point cloud without occluded points. The second and the fourth row are the new object point cloud with occluded points. It is obvious that the object point cloud with occluded points is more complete than the raw object point cloud.

Deep Learning Based Point Cloud Classification Approach
We choose to use PointNet as the classification approach. PointNet [2] proposed by Qi et. al, is an method that directly processes the original point cloud. Point-Net mainly consists of several transformation layers and several Multi-Layer Perceptron (MLP) blocks. The first layer of PointNet takes n points as input and learns a D × D transformation matrix through the T-Net learning, where D represents the feature dimension. The transformed data then goes through several Multi-Layer Perceptron(MLP) blocks shared by each point, an intermediate max pooling layer, a spatial transformation layer and two fully connected layers. The initial value of the spatial transformation matrix is set to an identity matrix. Except for the last layer, ReLU and Batch Normalization are applied to all other layers.
MLP of PointNet is implemented by the convolution of shared weights. The convolution kernel of the first layer is 1 × 3, and the subsequent convolution kernel size is 1 × 1.

Deep Learning Based Point Cloud Classification Approach With Occlusion Modeling
Based on the original PointNet network, we make some modifications to utilize the occlusion property proposed in this paper. We expand the input data from 3D to 4D, i.e. n × 4 in order to enable PointNet to process new formats of point cloud data. For the D × D transformation matrix obtained by the T-Net learning, we have also modified them so that the feature dimension of the new transformation matrix becomes 4 × 4.
In the subsequent module, we have also made appropriate modifications to the network. The size of the convolution kernel of the MLP is modified to 1 × 4 according to the input data dimension, and the output dimension of the last layer is set to the number of classes. Changed parts are shown in the bottom bounding box. We can see that we do not need to make many changes on the structure of the network itself. Our approach could be applied to any network which can directly process the raw lidar point cloud data.

Experimental Results
We divide our experiments into two parts and we choose to perform experiments on the KITTI dataset. We firstly did experiments on the seven categories ('car', 'van', 'truck', 'pedestrain', 'cyclist', 'tram' and 'misc') of KITTI dataset. As 'car', 'van' and 'truck' share a lot of similarities, and in fact they all belong to the 'vehicle' category, we then merge car, van and truck to a single category, and perform experiments on these five categories.

Classification Results on the 7 Categories
We separately train the PointNet network on the original point cloud and the point cloud with occluded points. The classification results are shown in Table.1 and Fig.6. Experimental results show that both the overall accuracy and perclass accuracy of our approach have a significant improvement compared with the original PointNet. In Fig.7, we show the confusion matrix of the original PointNet and our approach. In Fig.8, we show the comparision between the point cloud with and without the added points. For many samples occluded by obstacles, their incomplete point cloud always result in wrong classification, such as sample C in Fig.8. Due to the incompleteness of the point cloud, sample C is classified as 'misc' category in the original PointNet. In our approach, with the help of the added occluded points, it is correctly classified as the 'car'.

Classification Results on the 5 Categories
We merge car, van and truck into a single class and perform the experiments on the five categories. We believe that these three categories all belong to the 'vehicle' class, and they are equally important to the self-driving cars. The classification results are shown in Table.3 and Fig.10.   In Fig.10, it is easily seen that each category's classification accuracy of our approach is improved in our approach. Some qualitative examples are shown in Fig.9. The confusion matrix is shown in Fig.11.

Concluding Remarks
In this paper, we investigate the lidar classification problem in occluded scenarios. We model occlusion as a intrinsic property of the lidar point cloud, and add a pre-precessing step to the lidar point cloud processing pipeline. It is important to emphasize that our approach is not only applicable to enhance PointNet's classification performance. We believe that our approach for modeling occlusion is an important pre-processing step that can enhance any classification approaches.