{'fraction_non_alphanumeric': 0.02938695163104612, 'fraction_numerical': 0.01061586051743532, 'mean_word_length': 5.012256973795435, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improvement of projection-based LiDAR data segmentation algorithms using object-contextual representations

Nowadays self-driving cars and such unmanned aerial vehicles as drones are one of the most actively developed technologies, where machine learning algorithms play an irreplaceable role especially in the perception problem, which is the context of this research. To be applicable in self-driving cars and especially drones such algorithms should not only have good output quality, but also be real-time. For this reason, in the case of LiDAR data segmentation problem we pay special attention to algorithms that are based on point cloud projections because of their speed superiority over other heavy algorithms that process input point cloud directly. The main drawback of projection-based algorithms is their lower segmentation accuracy, so in this paper we show that it can be improved by integrating contextual representation module inside segmentation algorithm architecture. In our work we consider SalsaNext as a segmentation algorithm and OCR as a context representation module because these methods are among the highest in the corresponding datasets’ leaderboards. We provide results from quantitative evaluation on the Semantic-KITTI dataset, which demonstrate that the proposed SalsaNext modification gives 6.2% mean intersection over union metric improvement with no speed reduction.

1. Introduction Self-driving cars and such unmanned aerial vehicles as drones are currently among the most popular and rapidly developing technologies, and there is a very important problem of object recognition around an autonomous vehicle. The most common sensor for collecting such information is a stereo camera, but there are many examples of situations when a single camera may not be enough, such as fog, heavy rain, snowfall, and even at night time. Therefore, in order to correct these flaws, the car or drone is equipped with an additional perception device, namely a LiDAR.
For this reason, LiDAR point cloud semantic segmentation is one of the most important steps in the scene understanding problem that provides rich information about the autonomous vehicle's surrounding. This information is used for solving such self-driving problems as SLAM [1] and motion planning [2].
To be applicable in self-driving cars and drones, segmentation algorithms should have high accuracy with low latency at the same time. However, we cannot give arbitrary computing power to self-driving cars and especially drones, since at present this would not only significantly increase the cost of such systems, but it would also make it less useful in practical terms, since powerful hardware requires a lot Currently, there are two main deep learning approaches for LiDAR data segmentation (according to [3,4]): point-wise and projection-based neural networks. First approach operates directly on the raw 3D points or it's volumetric representation, whereas the second uses various point cloud flat projections.
And we pay special attention to LiDAR data segmentation algorithms that are based on point cloud projections (see figure 1 on the bottom) because of their superiority over other heavy algorithms that process input point clouds directly in terms of low latency. This transition from a 3D point cloud to a 2D image makes it possible to actively use the advantages of deep convolutional neural networks, which have proven themselves well for object detection [5], semantic [6] and instance segmentation [7].
The main drawback of projection-based algorithms is their slightly lower segmentation accuracy [3], so there were several attempts to deal with it in [8], and in this paper we show that it can be improved by integrating object-contextual representation module inside segmentation algorithm. In our work we consider SalsaNext [9] as a target segmentation algorithm and OCR [10] as a context representation module to integrate because they are among the highest in the corresponding datasets' leaderboards [11,12]. We address segmentation problem in the case of self-driving cars, but all results presented here can be easily applied for such unmanned aerial vehicles as drones, equipped with LiDAR.
This paper discusses the problem of multi-beam 360-degrees LiDAR point cloud segmentation, which are widely used in robotics and unmanned vehicles. Such LiDARs usually generate 2048 points for each beam (horizontal resolution). The total number of beams (vertical resolution) can be 128, 64, 32, 16, etc. depending on the sensor model. Each point contains information about x, y, z, depth, and intensity of diffuse reflection on surfaces (remission).
It is required to investigate the possibilities of improving the quality and ensuring a high segmentation speed, which allows each point from the cloud to be assigned one of 19 categories: car, bicycle, motorcycle, truck, other-vehicle, person, bicyclist, motorcyclist, road, parking, sidewalk, otherground, building, fence, vegetation, trunk, terrain, pole, traffic-sign. These categories were selected because they are included in the very popular large-scale Semantic KITTI dataset [11].
To evaluate the results of our model we use the intersection-over-union (IoU) metric (or Jaccard Index) over all classes that is given by formula (1) [9]: where is the set of points with a class prediction , is the set of points with ground truth class and || means the cardinality of the set. So the aim of this paper is to integrate OCR [10] module inside SalsaNext [9] architecture and obtain quantitative experimental results in terms of speed and mean IoU metric improvement.

Methodology
In our work we consider SalsaNext [9] as a target projection-based segmentation algorithm because of its superiority over other state-of-the-art algorithms in terms of speed (see section 3). SalsaNext is a LiDAR point cloud semantic segmentation neural network, which is an improvement of the RangeNet++ [13] approach, whose main idea is to use a spherical projection as a lightweight representation of the original point cloud, that significantly reduces inference time. This projection has the same properties as a regular image, so this allows to apply standard methods such as convolution and pooling without any changes. SalsaNext has an encoder-decoder architecture, where the encoder part consists of a series of ResNet [14] blocks with dropout and pooling. The decoder part makes upsampling via transposed convolution and fuses features extracted in the residual blocks. Each convolution layer in the SalsaNext architecture uses leaky-ReLU activation function and batch normalization for internal covariate shift treatment. To deal with class imbalance problem SalsaNext uses weighted softmax cross-entropy loss. The input of SalsaNext is rasterized image of spherical projection (formula (2) where ( , ) denotes image pixel coordinates of the corresponding LiDAR point ( , , ), ℎ and denote height and width of the projected image, represents the range of each point as = { + + }, defines the sensor vertical field of view as = | | + | |, and with denote LiDAR's lower field of view and upper field of view respectively. As shown in the network architecture diagram (figure 2), at the very beginning there is a so-called context module, the purpose of which is to aggregate the context information in different image regions. It's performed by fusing a larger receptive field with a smaller one by adding 33 and 11 kernels right at the beginning of the network. This helps to capture the global context alongside with more detailed spatial information. But the drawback of such a module is that it does not take into account the feature information that accumulates in the network during the input image is passed through its layers. And because SalsaNext operates with a flat projection image, for solving this problem we can apply an OCR [10] approach that was designed to take into account exactly such features. We prefer to use OCR as one of the most powerful [12] context aggregation methods at the moment of writing this paper. Figure 2. Proposed modification of SalsaNext [9] architecture with two possible positions for OCR [10] module integration and excluded intensity channel from the input image shape. More precisely, OCR is intended to be used for refining coarse segmentation that can be extracted from the last layers of the backbone network. This allows us to take into account the context of each image pixel and produce more accurate segmentation. The key idea of OCR approach is to use objectcontextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, object regions are learned under the supervision of the ground-truth segmentation. Second, the object region representation is computed by aggregating the representations of the pixels lying in the object region. Last, the relation between each pixel and each object region is computed, and the representation of each pixel is augmented with the object-contextual representation which is a weighted aggregation of all the object region representations. OCR module architecture is presented on figure 3.

Results and discussion
At the first, we have made a comparison in terms of speed and mean IoU metric between the top 3 accuracy-efficient algorithms from the leaderboard of the large-scale Semantic KITTI dataset [11], contained over 43K annotated full 3D LiDAR scans, that is currently the standard dataset for evaluating the quality of outdoor scene segmentation models. Evaluation on the KITTI 08 sequence (validation split) showed that SalsaNext network [9], which is a projection-based algorithm, is at least 2x faster than Cylinder3D [15] with SPVNAS [16] and has 1.13x lower mean IoU (table 1), that corresponds to metrics from Semantic KITTI leaderboard. Therefore we took SalsaNext as the target network for our research. Table 1. Speed and accuracy comparison between top 3 point cloud segmentation algorithms on Semantic KITTI dataset [11]. Weights for tested models were provided by their authors. The evaluation was performed on the GPU Nvidia Tesla V100 32GB.

Metric
Cylinder3D In our case we use SalsaNext network as backbone, which provides coarse segmentation for OCR module. But if we try to apply the module in a standard way, i.e. at the end of the network, then we will face difficulties because this approach requires a lot of device memory that network runs on. For example, 32 GB memory space of GPU Nvidia Tesla V100 isn't enough to train such architecture in a reasonable time. Thus we made an attempt to apply the OCR module in the centre network layers, where we have an image with much smaller dimensions than on the last layer after a cascade of convolutions: parameters (x, y, z, depth) instead of 5 (x, y, z, depth, intensity) as in original paper. This was done to increase the generalization ability of SalsaNext by preventing overfitting to intensity values range of specific LiDAR, which allows us to test trained networks on datasets collected by LiDARs with different parameters without large accuracy loss ( figure 4).
Thus, in this paper we investigate three SalsaNext modifications (figure 2): • Default SalsaNext (with 4-channel input image); • SalsaNext with integrated OCR module at first position; • SalsaNext with integrated OCR module at second position. Figure 4. SalsaNext [9] segmentation comparison on the same point cloud scene that was collected with Velodyne HDL-32 LiDAR: left -network was trained on Semantic KITTI dataset [11] (Velodyne HDL-64 LiDAR) with intensity channel, right -trained on the same dataset without intensity channel.
We trained all three SalsaNext modifications on the same Semantic KITTI dataset for 80 epochs with learning rate equal to 0.05 and other default parameters derived from the original paper [9]. We follow exactly the same protocol as source SalsaNext paper [9] and divide the dataset into training, validation, and test splits. For training we used sequences from 00 to 10, which contain over 21K scans. For validation we used sequence 08 (over 4K scans).
Best models' results of evaluation on the 08 sequence are presented in the table 2. The computational experiments were performed on the workstation with GPU Nvidia Tesla V100 32GB.
As we see from evaluation results presented in the table 2, SalsaNext + OCR (I) and SalsaNext + OCR (II) modifications outperforms default SalsaNext architecture for 2.5% and 6.2% correspondingly by mean IoU metric, which confirms the hypothesis that the closer to the end of the network OCR module is located, the greater quality gain it gives. At the same time, this module doesn't reduce the model speed. Someone can think that the increase of 6.2% mean IoU on validation set is too small, but such a result is the property of this module, which, moreover, has an easily calculated upper limit of a possible quality gain, for more details see the original paper [10]. For this reason, this experiment can be considered as successful.
The proposed improvement can also be useful when implementing the considered point cloud segmentation model for the on-board computer vision system of robotic platforms or such unmanned aerial vehicles as drones, where high recognition quality with low delays in the algorithm execution play a key role.

Conclusion
In this paper we presented an idea of the object-contextual representations [10] module integration inside the projection-based point cloud segmentation algorithm based on SalsaNext [9] network architecture. We obtained results that show us such modification gives expected mean IoU improvement that has tendency to increase with closer to the end of the network OCR module is placed, which allows us to expect more accuracy gain in case of using OCR in traditional way i.e. with the last network layers. So this defines our further research direction, specifically, to find a way to use the OCR module most efficiently with limited hardware resources. Also, it should be noted that all results presented in this paper can be applied not only for self-driving cars but also for such unmanned aerial vehicles as drones, equipped with LiDAR.