{'fraction_non_alphanumeric': 0.03191322119163654, 'fraction_numerical': 0.012513755698789498, 'mean_word_length': 5.015888027236618, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation

This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. Directly employing previous distillation approaches yields inferior results due to the intrinsic challenges of point cloud, i.e., sparsity, randomness and varying density. To tackle the aforementioned problems, we propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level. Specifically, we first leverage both the pointwise and voxelwise output distillation to complement the sparse supervision signals. Then, to better exploit the structural information, we divide the whole point cloud into several supervoxels and design a difficulty-aware sampling strategy to more frequently sample supervoxels containing less-frequent classes and faraway objects. On these supervoxels, we propose inter-point and inter-voxel affinity distillation, where the similarity information between points and voxels can help the student model better capture the structural information of the surrounding environment. We conduct extensive experiments on two popular LiDAR segmentation benchmarks, i.e., nuScenes and SemanticKITTI. On both benchmarks, our PVD consistently outperforms previous distillation approaches by a large margin on three representative backbones, i.e., Cylinder3D, SPVNAS and MinkowskiNet. Notably, on the challenging nuScenes and SemanticKITTI datasets, our method can achieve roughly 75% MACs reduction and 2x speedup on the competitive Cylinder3D model and rank 1st on the SemanticKITTI leaderboard among all published algorithms. Our code is available at https://github.com/cardwing/Codes-for-PVKD.


Introduction
LiDAR semantic segmentation plays a vital role in the perception of autonomous driving as it provides per-point semantic information of the surrounding environment. With the advent of deep learning, plenty of LiDAR segmentation models have been proposed [14,25,26,37] and have dominated the leaderboard of many benchmarks [1,3]. However, the impressive performance comes at the expense of heavy computation and storage, which impedes them from being deployed in resource-constrained devices.
To enable the deployment of these powerful LiDAR segmentation models on autonomous vehicles, knowledge distillation [10] is a prevailing technique to transfer the dark knowledge from the overparameterized teacher model to the slim student network to achieve model compression. However, directly applying previous distillation algorithms [9,10,17,24,28] to LiDAR semantic segmentation brings marginal gains due to the intrinsic difficulty of point cloud, i.e., sparsity, randomness and varying density.
To address the aforementioned challenges, we propose the Point-to-Voxel Knowledge Distillation (PVD). As the name implies, we propose to distil the knowledge from both point-level and voxel-level. Specifically, to combat against the sparse supervision signals, we first propose to distil the pointwise and voxelwise probabilistic outputs from the teacher, respectively. The pointwise output contains finegrained perceptual information while the voxelwise prediction embraces coarse but richer clues about the surrounding environment.
To effectively distil the valuable structural knowledge from the unordered point sequences, we propose to exploit the point-level and voxel-level affinity knowledge. The affinity knowledge is obtained via measuring the pairwise semantic similarity of the point features and voxel features. However, straightforwardly mimicking the affinity knowledge of the whole point cloud is intractable since there are tens of thousands of points and the affinity matrix of these point features have over ten billion elements. Consequently, Comparison between the prediction of our method (PVD) and the competitive channel distillation (CD) algorithm [24] on SemanticKITTI validation set. Points that are mistakenly classified are painted red for better visualization. It is obvious that PVD can make the student model predict more accurately for those minority classes (person) and faraway objects (bicycles, highlighted with green rectangles) than the baseline distillation approach. Here, KD denotes knowledge distillation.
we put forward the supervoxel partition to divide the whole point cloud into a fixed number of supervoxels. At each distillation step, we only sample K supervoxels and distil the affinity knowledge computed from point features and voxel features in these supervoxels, thus significantly enhancing the learning efficiency. Considering that uneven numberof-points distribution exists among different classes and objects at distinct distances, we further introduce a difficultyaware sampling strategy to more frequently sample supervoxels that contain minority classes and faraway objects, emphasizing the learning on hard cases. We summarize our contributions as follow. To our knowledge, we are the first to study how to apply knowledge distillation to LiDAR semantic segmentation for model compression. To address the difficulty of distilling knowledge from point cloud, namely sparsity, randomness and varying density, we propose the point-to-voxel knowledge distillation. In addition, we put forward the supervoxel partition to make the affinity distillation tractable. A difficultyaware sampling strategy is also employed to more frequently sample supervoxels that contain minority classes and distant objects, thus remarkably enhancing the distillation efficacy on these hard cases. As can be seen from Fig. 1, our method produces much more accurate predictions than the baseline distillation approaches especially for those minority classes and faraway objects.
We conduct extensive experiments on the nuScenes [3] and SemanticKITTI [1] datasets and the results demonstrate that our algorithm consistently outperforms previous distillation approaches by a large margin on three contemporary models, i.e., Cylinder3D [36,37], SPV-NAS [25] and MinkowskiNet [5]. Notably, on the nuScenes and SemanticKITTI benchmarks, PVD achieves approximately 75% MACs reduction and 2× speedup on the top-performing Cylinder3D model with very minor performance degradation.

Related Work
LiDAR semantic segmentation: LiDAR semantic segmentation [5,6,8,14,18,21,22,25,26,[29][30][31]35,37] is crucial for the navigation of autonomous vehicles. PointNet [21] is one of the pioneering work that uses Multi-Layer Perception (MLP) to process point cloud directly. Although effective in processing small-scale point cloud, PointNet and its variants [22] are extremely slow to handle large-scale outdoor point cloud. To cope with large-scale outdoor point cloud, Hu et al. [14] exploit random sampling for point selection and a local feature aggregation module is designed to further preserve the key features. Xu et al. [31] put forward the range-point-voxel fusion network to make use of the advantages of different views. Zhu et al. [37] propose the Cylinder3D method that adopts the cylindrical partition and asymmetrical convolution to better employ the valuable information in point cloud. Tang et al. [25] leverage neural architecture search to automatically find the optimal structure for the task at hand. Although these models have shown impressive performance on various benchmarks, a common drawback is that these networks are too cumbersome to be deployed on resource-constrained devices. To enable the deployment of these powerful yet lumbersome LiDAR semantic segmentation models on real-world applications, we propose the point-to-voxel knowledge distillation to achieve model compression. Knowledge distillation: Knowledge distillation (KD) stems from the seminal work of G. Hinton et al. [10]. The primary objective of KD is to transfer the rich dark knowledge from a cumbersome teacher model to a compact student model to mitigate the performance gap between these two models. The majority of the KD approaches concentrate on image classification tasks and they take various forms of knowledge as the distillation targets, e.g., intermediate outputs [13,23], visual attention maps [12,33], inter-layer similarity maps [32], sample-level similarity maps [20,27], etc. Recently, some researchers have adapted conventional KD techniques to distil knowledge for semantic segmentation tasks [9,11,17,24,28]. For instance, Liu et al. [17] propose to distil three levels of knowledge simultaneously, namely the pixel-level knowledge, the pairwise similarity knowledge and the holistic knowledge. He et al. [9] make the student mimic the compressed knowledge as well as the affinity information of the teacher. Although previous distillation algorithms have shown excellent performance on 2D segmentation, straightforwardly deploying them on LiDAR segmentation tasks brings marginal gains owing to the inherent sparsity, randomness and varying density of point cloud. And to our best knowledge, we are the first to apply knowledge distillation to LiDAR semantic segmentation. The proposed PVD can effectively transfer both point-level and voxel-level knowledge to students and is suitable for distilling various LiDAR semantic segmentation models.

Methodology
Given an input point cloud X ∈ R N ×3 , the objective of LiDAR semantic segmentation is to assign a class label l ∈ {0, 1, ..., C − 1} to each point, where N is the number of points and C is the number of classes. Contemporary algorithms use CNNs for end-to-end prediction.
Considering that autonomous vehicles typically have limited computation and storage resources and call for realtime performance, efficient models are employed to fulfill the preceding requirement. Knowledge distillation [10] is widely adopted to achieve model compression via conveying the rich dark knowledge from the large teacher model to the compact student network. However, previous distillation methods [9,17,24,28] are tailored for 2D semantic segmentation tasks. Directly applying these algorithms to distill knowledge for 3D segmentation tasks produces unsatisfactory results owing to the intrinsic difficulty of point cloud, i.e., sparsity, randomness and varying density. To address the aforementioned challenges, we propose the Point-to-Voxel Knowledge Distillation (PVD) to transfer the knowledge from both the point level and voxel level.

Framework overview of Cylinder3D
We first have a brief review of the Cylinder3D model [37] and then introduce the point-to-voxel distillation algorithm based on it. Taking the point cloud as input, Cylinder3D first generates the corresponding feature for each point using a stack of MLPs and then reassigns the point features F p ∈ R N ×C f based on the cylindrical partition where C f is the dimension of point features. Point features that belong to the same voxel are aggregated together through the maxpooling operation to obtain the voxel features F v ∈ R M ×C f where M is the number of non-empty voxels. Next, these voxel features are fed into the asymmetrical 3D convolution networks to produce the voxel-wise output O v ∈ R R×A×H×C . A point-wise refinement module is further employed to produce the refined point-wise prediction O p ∈ R N ×C . Here, N , C, R, A and H denote the number of points, number of classes, radius, angle and height, respectively. Eventually, we will use the argmax operation to process the pointwise prediction to obtain the classification result of each point.

Point-to-Voxel Output Distillation
The primary difference between 2D and 3D semantic segmentation lies in the input. Compared with images, point cloud is sparse and it is difficult to train the efficient student model using the sparse supervision signal. Previous distillation approaches [10,17] typically resort to distilling the ultimate output of the teacher network, i.e., the pointwise output of the teacher network for LiDAR semantic segmentation. Although the pointwise output contains finegrained perceptual information of the environment, such knowledge is inefficient to learn as there are hundreds of thousands of points. To improve the learning efficiency, in addition to the pointwise output, we propose to distil the voxelwise output as the number of voxels is smaller and is easier to learn. The combination of both pointwise output distillation and voxelwise output distillation naturally form the coarse-to-fine learning process. The pointwise and voxelwise output distillation loss is given below: where KL(.) denotes the Kullback-Leibler divergence loss. Labels for the voxelwise output: Since a voxel may contain points from different classes, how to assign the proper label to the voxel is also crucial to the performance. Following [37], we adopt the majority encoding strategy that uses the class label having the maximum number of points inside a voxel as the voxel label.

Point-to-Voxel Affinity Distillation
Distilling the knowledge of the pointwise and voxelwise outputs is insufficient as it merely considers the knowledge of each element and fails to capture the structural information of the surrounding environment. Such structural knowledge is vital to the LiDAR-based semantic segmentation model as the input points are unordered. A natural remedy is to adopt the relational knowledge distillation [17] which calculates the pairwise similarity of all point features. However, there exist two shortcomings in this learn- We take the Cylinder3D model [37] as example. There are two networks in our framework, one is the teacher and the other is the student. The student model is obtained via pruning 50% channels of each layer of the teacher. A teacher model is comprised of five parts, i.e., point feature extraction module, point-to-voxel transformation module (voxelization), an encoderdecoder model (asymmetric 3D convolution network), the Dimension-Decomposition Contextual Modeling (DDCM) module, and the point refinement module. Given the input point cloud, we first divide it into a fixed number of supervoxels and sample K supervoxels according to the difficulty-aware sampling strategy (K=1 in this figure, denoted by the red sector). Then, the student is forced to mimic two levels of knowledge from the teacher. The first-level knowledge is the pointwise output and voxelwise output of the whole point cloud.
The second-level knowledge is the inter-point affinity matrix and inter-voxel affinity matrix of the sampled supervoxels.
ing scheme: 1) since there are usually hundreds of thousands of points in an input point cloud, the similarity matrix which has over ten billion elements is computationally expensive to calculate and extremely difficult to learn. 2) There exist significant quantity differences between different classes and objects at different distances. The abovementioned learning strategy ignores such difference and treats all classes and objects equally, thus making the distillation process sub-optimal. Supervoxel partition: In order to more efficiently learn the relational knowledge, we divide the whole point cloud into several supervoxels whose size is R s × A s × H s . Each supervoxel is comprised of a fixed number of voxels and the total number of supervoxels is ⌉ is the ceiling function. We will sample K supervoxels to perform the affinity distillation. Difficulty-aware sampling: To make supervoxels that contain less frequent classes and faraway objects more likely to be sampled, we present the difficulty-aware sampling strategy. The weight for choosing the i-th supervoxel is: where f class is the class frequency, d i is the distance of the outer arc of the i-th supervoxel to the origin in the X-Y plane. We treat the classes that have more than 1% of all points in the whole dataset as majority classes and the remaining classes are considered as minority classes. We empirically set the class frequency of the supervoxel as: f class = 4 exp(−2N minor ) + 1, where N minor is the number of voxels of minority classes in the supervoxel. If there is no voxel of minority class, the f class will be 5. And as the the number of voxels of minority classes increases, f class will be close to 1 quickly. Then, we normalize the weight and obtain the probability of the i-th supervoxel being sampled is: Wi . Point/voxel features processing: Note that for each point cloud, the number of input points is different and the den-sity is varying, thus making the number of point features and voxel features variable in a supervoxel. As to the calculation of loss function, it is desirable to keep the number of features fixed. Hence, we set the number of retained point features and non-empty voxel features as N p and N v , respectively. If the number of point features is larger than N p , then we will retain N p point features by randomly discarding additional point features of majority class. If the number of point features is smaller than N p , we will append all-zero features to the current features to obtain N p features, as is shown in Fig. 3 (a). The voxel features are processed in a similar way.
Here we have N p point featuresF p r ∈ R Np×C f and N v voxel featuresF v r ∈ R Nv×C f in the r-th supervoxel, respectively. Then, for each supervoxel, we calculate the inter-point affinity matrix according to the following equation: The affinity score captures the similarity of each pair of point features and it can be taken as the high-level structural knowledge to be learned by the student. The interpoint affinity distillation loss is given as below: The inter-voxel affinity matrix is computed similarly. Eventually, we make the student mimic the generated affinity matrices of the teacher model. The inter-voxel affinity distillation loss is presented as follows: Visualization of the learned affinity maps: From Fig. 4, we can see that PVD causes a closer affinity map between student and teacher. With PVD , features that belong to the same class are pulled closer while those of different classes are pushed apart in the feature space, resulting in a much clear affinity map. Compared with the rival channel distillation approach [24], PVD can better transfer the structural knowledge from teacher to student, which strongly validates the superiority of PVD in distilling LiDAR segmentation models.

Final objective
Our final loss function is composed of seven terms, i.e., the weighted cross entropy loss for the pointwise output and voxelwise output, the lovasz-softmax loss [2], the point-tovoxel output distillation loss and the point-to-voxel affinity distillation loss: where α 1 , α 2 , β 1 and β 2 are the loss coefficients to balance the effect of the distillation losses on the main task loss.

Experiments
Datasets. Following the practice of Cylinder3D [37], we conduct experiments on two popular LiDAR semantic segmentation benchmarks, i.e., nuScenes [3] and Se-manticKITTI [1]. For nuScenes, it consists of 1000 driving scenes, in which 850 scenes are selected for training and validation, and the remaining 150 scenes are chosen for testing. 16 classes are utilized for LiDAR semantic segmentation after merging similar classes and eliminating infrequent classes. For SemanticKITTI, it is comprised of 22 point cloud sequences, where sequences 00 to 10, 08 and 11 to 21 are used for training, validation and testing, respectively. A total number of 19 classes are chosen for training and evaluation after merging classes with distinct moving status and discarding classes with very few points. Evaluation metrics.
Following [37], we adopt the intersection-over-union (IoU) of each class and mIoU of all classes as the evaluation metric. The calculation of IoU is: T Pi+F Pi+F Ni , where T P i , F P i and F N i represent the true positive, false positive and false negative of class i. Implementation details. Following [37], we leverage Adam [15] as the optimizer and the initial learning rate is set as 2e-3. Batch size is set as 4 and the number of training epochs is 40. α 1 , α 2 , β 1 and β 2 are set as 0.1, 0.15, 0.15 and 0.25, respectively. We take the rival and open sourced Cylinder3D 2 [37] approach as the backbone since the topperforming RPVNet [31] and AF2S3Net [4] do not release their codes. Random flipping, rotation, scaling and transformation are taken as the data augmentation strategy. The size of the voxel output is 480×360×32, where the three dimensions denote the radius, angle and height, respectively. The size of the supervoxel is set as 120×60×8. N v and N p are set as 3000 and 6000, respectively. The number of sampled supervoxels K is set as 4. The inter-point affinity distillation is performed on the output of the point feature extraction module and the inter-voxel affinity distillation is conducted on the output of the encoder-decoder backbone. For nuScenes, Cylinder3D 0.5× is produced from the original Cylinder3D model by pruning 50% channels for each layer of the whole network. For SemanticKITTI, Cylinder3D 0.5× is obtained by merely pruning 50% channels for each layer of the asymmetrical 3D convolution network and we keep the point feature extraction module unchanged as it is vital to extract rich information from the input point cloud. We also apply our method to compress SPVNAS 3 [25] and MinkowskiNet [5] to verify the scalability of our algorithm. More details are provided in the supplementary material. Baseline distillation algorithms. In addition to the stateof-the-art methods in each benchmark, we also compare our method with classical KD methods and contemporary distillation approaches tailored for 2D semantic segmentation, including vanilla KD [10], SKD [17], CD [24], IFV [28] and KA [9]. Here, SKD takes the output probability maps and pairwise similarity maps as mimicking targets. We remove the original holistic distillation loss for SKD as incorporating GANs into current framework will cause severe training instability; CD utilizes the intermediate feature maps and score maps as knowledge; IFV transfers the intra-class feature variation from the teacher to the student; KA makes the student distil the compressed knowledge and the affinity information of the whole output from the teacher.

Results
Comparison with state-of-the-art LiDAR segmentation models: We compare our model with contemporary Li-DAR semantic segmentation models, e.g., KPConv [26], TORNADONet [7] and SPVNAS [25]. From Table 1, we can see that Cylinder3D 0.5×+PVD (the penultimate row) achieves comparable performance with the original Cylin-der3D model on the SemanticKITTI test set. Compared to KPConv and SPVNAS, our Cylinder3D 0.5×+PVD not only achieves better performance, e.g., 3.8% higher than SPVNAS in mIoU, but also has much lower latency than the SPVNAS method (259 ms v.s. 76 ms). Specifically, on minority classes such as bicycle, motocycle and bicyclist, the IoU of Cylinder3D 0.5×+PVD is at least 10.5% higher than the SPVNAS method. And with some engineering tricks like finetuning and flip & rotation test ensemble, our Cylinder3D 0.5×+PVD (the last row) can obtain 71.2 mIoU, which is 2.3 mIoU higher than the original Cylin-der3D model. Impressive performance is also observed in nuScenes validation set. Our Cylinder3D 0.5×+PVD exhibits similar performance with the original Cylinder3D network in terms of the overall mIoU and the IoU on each class. Comparison with previous distillation methods: From Table 1 and 2, we can see that PVD significantly outperforms baseline distillation algorithms in both benchmarks. The performance gap between PVD and the most competitive KD method is larger than 1.8. For instance, on Se-manticKITTI test set, our PVD is 2.8 mIoU higher than the CD method. And on both majority classes and minority classes, our PVD significantly outperforms traditional distillation algorithms. For instance, on nuScenes dataset, PVD is at least 2 mIoU higher than SKD in classes such as bicycle, bus, car, trailer and sidewalk. The aforementioned results strongly demonstrate the effectiveness of PVD in transferring knowledge for teacher-student learning. Generalization to more architectures: To verify the generalization of our method, we also apply PVD to compress SPVNAS [25] and MinkowskiNet [5]. Since SPVNAS does not provide the training code for the NAS-based architecture, we conduct experiments on its manually designed architecture. As can be seen from Table 3, our PVD still brings more gains to the student model than baseline distillation algorithms. For instance, our PVD outperforms the SKD algorithm by 2.6 mIoU in terms of mIoU on the SPVNAS backbone. It is noteworthy that PVD can safely achieve 75% MACs reduction without causing severe performance drop. The above results strongly demonstrate the good scalability of our method. Qualitative results: As can be seen from Fig. 5, compared with the SKD approach, our PVD greatly improves the prediction of the student model. The prediction errors of PVD on those minority classes, e.g., person and bicycle, are significantly smaller than those of SKD. Besides, on objects that are faraway from the origin, e.g., the car highlighted by the green rectangle, PVD also yields more accurate predictions than SKD. And PVD has lower inter-class similarity and higher intra-class similarity, which explicitly showcases

Ablation studies
In this section, we perform comprehensive ablation studies to examine the efficacy of each component, supervoxel size as well as the sampling strategy on the final  Figure 5. Visual comparison of different methods on the SemanticKITTI validation set. Here, the ground-truth for the inter-voxel affinity map is the ideal map where the intra-class similarity score is 1 and inter-class similarity score is 0.  Figure 6. Comparison between different sampling strategies.
Supervoxel size. Note that we perform the inter-point and inter-voxel distillation on the sampled supervoxels. Supervoxel size has a non-negligible effect on the efficacy of PVD since a overly small size will make student learn little from the affinity distillation loss while a large supervoxel size will weaken the learning efficiency of PVD. Here, we keep the number of sampled supervoxels as 4 to remove the effect of this factor. From Table 5, we can see that setting the supervoxel size to (120, 60, 8) yields the best performance. Remarkably increasing or decreasing the supervoxel size will harm the distillation efficacy. Sampling strategy. We compare four different sampling strategies, i.e., the original difficulty-aware sampling, distance-aware sampling, category-aware sampling and random sampling. Here, distance-aware sampling is to only more frequently sample distant points while category-aware sampling will more likely sample points belonging to rare classes. From Fig. 6, it is apparent that difficulty-aware sampling brings more gains than the other three strategies. Specifically, difficulty-aware sampling outperforms both distance-aware and category-aware sampling, suggesting that both distance and categorical awareness are crucial to the distillation effect. The large gap between difficultyaware sampling and random sampling validates the necessity of difficulty-aware sampling strategy.

Conclusion
In this paper, we propose a novel point-to-voxel knowledge distillation approach (PVD) tailored for LiDAR semantic segmentation. PVD is comprised of the point-tovoxel output distillation and affinity distillation. The supervoxel partition and difficulty-aware sampling strategy are further proposed to improve the learning efficiency of affinity distillation. We perform experiments on two LiDAR semantic segmentation benchmarks and show that PVD significantly outperforms baseline distillation algorithms on distilling Cylinder3D, SPVNAS and MinkowskiNet. The impressive results indicate that there still exists large redundancy in 3D segmentation models and our approach can serve as a strong baseline to compress these cumbersome models.

A. Quantitative results
We provide the complete quantitative results of different algorithms on SPVNAS [25] and MinkowskiNet [5] in Table 6. Apparently, PVD consistently outperforms previous distillation algorithms by a large margin. For instance, on SPVNAS, PVD can bring 2.6 more points than the SKD method in mIoU. For both models, PVD can almost mitigate the performance gap between the original network and the pruned model. The encouraging results on SPVNAS and MinkowskiNet convincingly demonstrates the good scalability of PVD.

B. Ablation studies
Loss coefficients. By comparing each row with the last row in Table 7, we have the following observations: 1) the loss coefficient of the inter-voxel affinity distillation should be larger than other distillation losses to yield the best distillation effect (row 1 and 2). 2) Exchanging the loss coefficients of the point-based distillation loss and voxel-based distillation loss deteriorates the performance, which means the voxel-based distillation loss guides the point-based loss and is more important (row 5). 3) Slightly increasing the loss coefficients will not significantly affect the overall performance, which demonstrates the robustness of PVD (row 3 and 4).
Performance sensitiveness to f class . We conduct experiments on examining the effect of f class . We rewrite the f class to be f class = α exp (βN minor ) + 1. Then, we randomly choose α from {3, 4, 5, 6} and β from { -1, -2, -3}, and compare the performance of different combinations. Experimental results reveal that the final performance

C. Elaborated implementation details
For the baseline knowledge distillation approaches, the value of each loss coefficient is provided in Table 9. Since training a single model from scratch may take one more week, we resort to loading the pre-trained weights to accelerate the training process. In this condition, the overall training duration will be shortened to three days. Note that all methods adopt this strategy to ensure fair comparison. The latency is recorded using a single GPU (NVIDIA Tesla PG503-216 GV100) and the final value of latency is obtained after averaging the latency of 100 samples. The training protocol for SPVNAS and MinkowskiNet is exactly the same as their open-sourced codes 5 . Finetuning denotes retraining the trained model for 10 more epochs with the learning rate being 2e-4. MACs calculation: Since sparse convolution merely operates on the non-zero positions and different input point cloud sequence has different non-zero patterns, we first estimate the average kernel map size following [25] for each layer and then use the following equation to compute the FLOPs of each layer: F LOP s = N × K s × C in × C out , where K s is the size of kernel map, N is the number of points, C in is the number of input channels, C out is the number of output channels.

D. Qualitative results
We provide more visual comparisons of PVD with previous distillation algorithms in Fig. 7. Compared with the rival SKD approach, our PVD can significantly improve the prediction of the student model. The prediction errors of PVD on those minority classes, e.g., person and bicycle, are much smaller than those of SKD. Besides, on objects that are faraway from the origin, e.g., the car highlighted by the green rectangle, PVD also yields more accurate predictions than SKD. And PVD has lower inter-class similarity and higher intra-class similarity than SKD. The aforementioned results explicitly demonstrate the efficacy of PVD in distilling structural knowledge from the teacher model to the student.  Figure 7. Visual comparison of different methods on the SemanticKITTI validation set. Here, the ground-truth for the inter-voxel affinity map is the ideal map where the intra-class similarity score is 1 and inter-class similarity score is 0.