{'fraction_non_alphanumeric': 0.03620945516217935, 'fraction_numerical': 0.01666313334746661, 'mean_word_length': 4.939561823218333, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LMSCNet: Lightweight Multiscale 3D Semantic Completion

We introduce a new approach for multiscale 3D semantic scene completion from sparse 3D occupancy grid like voxelized LiDAR scans. As opposed to the literature, we use a 2D UNet backbone with comprehensive multiscale skip connections to enhance feature flow, along with 3D segmentation heads. On the SemanticKITTI benchmark, our method performs on par on semantic completion and better on completion than all other published methods - while being significantly lighter and faster. As such it provides a great performance/speed trade-off for mobile-robotics applications. The ablation studies demonstrate our method is robust to lower density inputs, and that it enables very high speed semantic completion at the coarsest level. Qualitative results of our approach are provided at http://tiny.cc/lmscnet.


Introduction
Understanding 3D surroundings is a natural ability for humans. While past experience allows us to reason about scene geometry and semantics of an entire scene, this proves difficult for computers given the inherently sparse nature of 3D sensors [2] (due to sparse sensing, limited field-of-view, and occlusions). Still, a comprehensive 3D sensing of the scene is crucial for applications like mobile robotics, and more especially for autonomous driving. Recently, semantic scene completion was proposed [36] as a new generative task, where both completion and semantic labels are inferred for the whole scene.
Unlike images, conveniently encoded as 2D tensors, 3D data causes representation challenges. It is thus common to encode the latter as voxel grids processed by 3D Convolutional Neural Networks (CNNs) [11,36,12,26]. This shows good results but also requires heavy computation, as the memory requirement grows cubically with the input voxel resolution [28]. Consequently, most of the literature limits the predicted resolution and network depth, being incapable to perform the task at the same spatial resolution as the input [36,15,40]. This drawback has limited the deployment of such methods for real time applicationsi.e. augmented and virtual reality [38], robotics perception and navigation [23], scene understanding [25], among others -that would greatly benefit of semantic scene completion from sparse LiDAR scan.
We tackle this problem, and propose a Lightweight Multiscale Semantic Completion, coined LMSCNet, where a 3D voxel grid is processed with considerably lighter 2D convolutions without need of additional modalities [1,15]. This is achieved by convolving along one spatial axis (close in spirit to bird-eye view process [5]), while mapping to third dimension with 3D segmentation heads. In our proposal, multiscale completion is also possible given informative features map flow, preserving computation efficiency and enabling very fast inference at coarse levels. Fig. 1 shows the multiscale output of our LMSCNet on the Se-manticKITTI dataset [1], using a single sparse LiDAR scan input encoded as voxel. While some works use progressive multiscale losses [25,10,12], the literature ignores the benefit of multiscale completion which we prove useful for reducing inference times. To summarize, the main contributions of our work are: • a novel 3D semantic scene completion pipeline using an occupancy grid, • a lightweight architecture with mix of 2D/3D convolutions leading significantly less parameters, • a modular multiscale pipeline which allows coarser inference at very high speed, • state of the art performance on SemanticKITTI [1] and better performance on completion.

Related Works
To process 3D data such as point-clouds, some use birdeye-view [6] or 2D spherical [29] projection. Still, the common strategy relies either on point [32] or voxel [33] networks. The inherent limitation of voxel representation is the staggering memory requirement due to empty voxels, which led to optimized structures [33] or use of sparse convolutions [18] to prevent dilation of the data manifold. When it comes from real sensing, 3D data is inherently sparse (e.g. LiDAR scan, stereo, etc.) and its densification was initially framed as a completion or reconstruction task. Recent works though, also assign semantic labels to their output subsequently referring to this as semantic completion.
More recently, learning methods boosted the completion of occluded and unseen regions [14,17,42]. In [22], voxel labels are predicted using a Conditional Random Field (CRF), while [11] uses 3D-Encoder-Predictor to correlate observed scene with a priori known 3D shapes. A few works also benefit from Signed Distance Functions representation as it provides richer gradient flow [31,33,10]. For memory reason, [10] uses TSDFs input with sparse encoder and partially dense decoder to propagate features in unknown regions. While this effectively reduces memory, because TSDFs are denser than occupancy grids, we argue it would require a bigger and denser decoder, thus annihilating the benefit of any sparse encoding. Other end-to-end completion networks were also proposed in [9,4].
3D Semantic Scene Completion. SSCNet [36] was the first work to combine semantic segmentation and scene completion end-to-end with 3D CNNs. Further works also use additional RGB data by projecting or fusing semantic features from an image network [15,26,24,27]. An alternative to 3D data only is to encode LiDAR scans as spherical projection [29], which enriches neighboring information [1,19]. While this boosts performance, it also increases the network complexity and subsequently the inference time. Generative Adversarial Networks (GANs) have also been proposed to enforce realistic outputs [39,7] but are harder to train. To lower memory consumption with the preferred voxelized representations, Spatial Group Convolutions (SGC) [40] divide input into groups for efficient processing at the cost of small performance drops.
Different from the literature, we rely solely on 3D voxelized occupancy data avoiding any preprocessing (as for TSDFs, SDFs, etc.), and propose a lightweight architecture with additional multiscale capability.

Method
We tackle the problem of dense 3D semantic completion where the task is to assign a semantic label to each individual voxel. Given a sparse 3D voxel grid, the goal is to predict the 3D semantic scene representation, where each voxel is being assigned a semantic label C = [c 0 , c 1 , . . . , c N ], where N is the number of semantic classes and c 0 stands for free voxels.
Our architecture, coined LMSCNet and shown in Fig. 2, uses a lightweight UNet style architecture to predict 3D semantic completion at multiple scales, allowing fast coarse inference, beneficial for mobile robotics applications. Instead of greedy 3D convolutions, we mostly employ 2D convolutions along the height axis; similar to a bird-eye view. In the following we detail our custom lightweight 2D/3D architecture (Sec. 3.1), the multiscale reconstruction (Sec. 3.1), and the overall training pipeline (Sec. 3.2).

Lightweight multiscale 2D/3D architecture
To infer a dense output from the sparse input voxel grid, we use a standard encoder-decoder UNet architecture with 4 levels, thus learning features at decreasing resolutions. At each level, a series of convolution operations is applied followed by a pooling; downscaling the resolution size by 2. The reduction of spatial dimensions in UNets is beneficial for semantic tasks as it subsequently increases the kernels field-of-view at no cost. Note that dilated convolutions Our pipeline uses a UNet architecture with 2D backbone convolutions (in blue) and 3D segmentation heads (in gray) to perform 3D semantic segmentation and completion at different scales, while preserving low complexity. Convolution parameters are shown as: (number of filters, kernel size and stride). Notice that we intentionally lower the 2D features dimension and use Atrous 3D convolutions (ASPP blocks from [26]) to preserve low inference complexity.
(a.k.a 'atrous') with increasing dilation rates cannot be used in the encoder due to the sparse input nature. Though dense convolutions in the encoder imply a dilation of the input manifold [18], we argue this is beneficial for 3D semantic completion, given the sparse →dense nature of the task.
2D backbone. To preserve a lightweight architecture, we use 2D convolutions along the X,Y dimensions, thus turning the height dimension (Z) into a feature dimensionsimilar in spirit to CNNs encoding point cloud as bird-eyeview [6]. While this implies loosing 3D spatial connexity, it also enables significantly lighter operations. To further reduce the memory requirements, we keep a minimum number of features in each convolution layer. Along with the standard skip connections, we also enhance information flow in the decoder by concatenating the output of each level to all lower levels. Technically, we upsample coarse feature maps learning ad-hoc deconvolution before concatenation to lower levels, which is shown with purple deconv blocks along blue and gray arrows in Fig. 2. Intuitively, this enables our network to use high level features from coarser resolutions, and thus enhancing the spatial contextual information.
3D segmentation head. Different from other works handling point cloud as bird-eye-view, the task of 3D semantic completion actually requires to retrieve the 3rd dimension "lost" with 2D convolutions. In other words, while 2D CNNs output 3D features maps, our decoder must output 4D tensor; the last dimension being the semantic class-wise probability distribution. To address this, we introduce 3D segmentation heads depicted as gray blocks in Fig. 2. The heads use a series of dense and dilated convolutions. The latter, in the form of Atrous Spatial Pyramid Pooling (aka ASPP [5,26]), is beneficial to fuse information from different receptive fields thanks to the convolutions with increasing dilation rates (here [1, 2 and 3]). Note that dilated convolutions, though light and powerful, are not appropriate for sparse inputs and, as such, cannot be used in the encoder. In our segmentation head, the benefit of preceding ASPP with dense 3D convolutions is dual: a) to further densify the feature maps, b) to ward off features from the segmentation heads and the backbone features. This last property is required to enable multiscale capacity, which we now describe.
Multiscale completion. In the same vein as [40,10], we aim to output multiscale completion to enable both coarse scene representation and faster scene completion at lower resolution -beneficial for mobile robotics applications. We subsequently attach a 3D segmentation head after each level of the 2D UNet architecture, thus providing outputs at input relative scale of 1 2 l ∀ l ∈ {0, 1, 2, 3}. A sample output is shown in Fig. 3. As already mentioned, we noticed experimentally the importance of separating the segmentation features from the main features of the 2D backbone, which again justifies the additional 3D convolutions in the segmentation head. The main interests of our multiscale architecture is that it infers semantic scene completion at a desired scale as needed, reducing the computation and memory requirements. This is further analyzed in Sec. 4.1.1.

Training strategy
We train our LMSCNet from scratch in a standard endto-end fashion from pairs of sparse input voxel (x) and semi-dense semantically labeled voxel grid (ŷ). It is impor- To supervise coarser representation, we use majority vote pooling from the original resolution ground truth.
tant to note that in a real setup, a dense ground truth is impractical for scene completion, due to occlusions and sensor field-of-view limitations. As such, the ground truthŷ is also sparse and encoded with N+2 classes (N semantic classes, 1 free class, 1 unknown). Similar to others [36,26,15] we use a sparse loss strategy, back propagating the gradient only where ground truth is known.
For each scale l, we train with a cross-entropy loss defined as where y is the network output, i a voxel index, andŷ i,c a one-hot vector (i.e.ŷ i,c = 1 if voxel i is labeled class c, otherwiseŷ i,c = 0). Note that semantic tasks are by nature highly class-imbalanced problems. This is especially true in outdoor settings, which causes the prevalence of classes like road or vegetation. We account for the class-imbalance nature in Eq. 1 by weighting each class loss according to the inverse of the class-frequency f c as in [29], thus using w c = 1 log (fc+ ) (with 1). Finally, the complete network loss is a weighted sum of all level losses 1 and writes: Note that some of our choices were guided by faster training or inference speed. For example, unlike [40,41,13,10,36] we avoid using flipped Truncated Signed Distance Function (fTSDF) that requires a greedy computation time and was found to be of little benefit [15,1]. We also tried to encode input as N+2 classes, that is with unknown class, but we noticed little improvement -if any -at the cost of a large pre-processing time for ray casting.

Experiments
We evaluate our LMSCNet method by training on the recent semantic scene completion benchmark Se-manticKITTI [1] providing 3D voxel grids from semantically labeled scans of HDL-64E rotating LiDAR in outdoor urban scenes [16]. In [1], inputs are voxelized single scans, while the ground truth was obtained from the voxelized aggregation of successive registered scans. Grids are 256x256x32 with 0.2m voxel size, and it is important to note that input and ground truth are sparse, with average density of 6.7% and 65.8%, respectively. We use standard mIoU as a semantic completion metric, measuring the intersection over union averaged over all classes (20 semantic classes + free). Additionally, we consider completion metrics IoU, Precision, and Recall to provide a sense of the scene completion quality, regardless of the assigned semantic labels (i.e. considering the binary free / occupied setting). Note that completion is crucial for obstacle avoidance in mobile robotics.
Implementation details. We train using the original train/val splits with 3834/815 grids [1], adding x-y flipping augmentation for generalization. Adam optimizer is used (β 1 = 0.9, β 2 = 0.999) with learning rate of 0.001 scaled by 0.98 epoch . Training fits in a single 11GB GPU with batch size 4, taking around 48 hours to converge ( 80 epochs).

Performance
In the following we report performance against four state-of-the-art methods: SSCNet [36], TS3D [15], TS3D+DNet [1], TS3D+DNet+SATNet [1]. Because SSC-Net output is 4x downsampled, we also report performance using deconvolution to reach full input resolution, hereafter denoted SSCNet-full. We refer to the supplementary for details on the required architectures adjustments. Hereafter, we denote our multiscale architecture as LMSCNet. We detail semantic completion performance and then demonstrate the speed and lightness of our architecture.

Semantic Scene Completion
Performance on the SemanticKITTI benchmark [1] is reported in Tab. 1 against all published methods and SSCNetfull. The evaluation was conducted on the official server (i.e. hidden test set) hence, with the full size ground truth.
Overall, we perform on par with the best methods, though 2nd on the semantic completion metric (mIoU). On the latter, TS3D+DNet+SATNet [1] is slightly better despite their significantly heavier and slower network. Note also that TS3D uses additional RGB input, and all TS3D+DNet use also LiDAR refraction intensity. Conversely, LMSC-Net is more versatile as it uses only occupancy grid input. Notice that the highly imbalanced class frequencies (shown in parenthesis in Tab. 1) also illustrate the task complexity. Specifically, we outperform others on the largest four classes but performs on par or lower on the others, which advocates for some improvement in our balancing strategy. On the completion metrics (IoU) our method outperforms  all others by a comfortable margin. Again, completion is of high importance for practical mobile robotics applications.
In addition to the multiscale proposal (LMSCNet), we also report LMSCNet-singlescale -a variation of LMSCNet where we train with L = L 0 -, which logically performs a little better at full size though at the cost of loosing crucial multiscale capacity.
Qualitative performance. We compare qualitatively full size outputs of our LMSCNet and SSCNet-full in Fig. 4, with views pairs from 4 scenes of the SemanticKITTI validation set 2 . At the rightmost, ground truth visualization also illustrates the sparse supervision complexity since holes are still visible. Our method produces visually smoother semantic labels, easily noticeable in rows 5-8, and is able to reconstruct thin structures, like trees or cars (rows 6 or 7). For comprehensive analysis, we further test the same model (trained on 64-layer LiDAR) on the popular nuScenes dataset [3], which has been registered using a 32layers lidar sensor. Fig. 5 shows that our network better adjusts to the change of density and maintains the smoothness in the reconstruction.
Multiscale performance. Tab. 2 shows multiscale performance of our method on the SemanticKITTI validation set, where the scale is relative to the full size resolution (level 0). From Sec. 3.1, scale at level l is 1 2 l . Ground truths at lower resolution were obtained from majority vote pooling of the full size ground truth. From the above table, our architecture maintains a good performance in all resolutions, with 2 Note that SemanticKITTI benchmark (i.e. test set) does not provide any visual results. Hence, we omit TS3D baselines due to retraining complexities and their use of additional modalities (RGB or LiDAR intensity).  Table 2: LMSCNet multiscale semantic completion performance on SemanticKITTI validation set. We reach good performance at all levels, even better at the coarsest levels.
best performance logically reached at the lowest resolution (highest level). Qualitative multiscale completion is visible in Fig. 3. We argue that our architecture reaches multiscale capacity thanks to the disentanglement of segmentation features with our custom head. Additionally, at coarser resolution our network reaches very fast inference, which will be described in details in the following section.

Architectures comparison.
Tab. 3 reports networks statistics for our architecture and all above mentioned baselines. From the latter, even at full size LMSCNet has significantly less parameters (0.35M) and lower computational cost for inference (72.6G FLOPs). Compared to any TS3D baselines it is at least an order of magnitude faster. However, SSCNet (original or full) is twice faster than LMSCNet, though with more parameters and slightly worse performance (cf. Tab. 1).
In last rows of Tab. 3, we report statistics for coarser completion, removing unnecessary parts of our network at inference. Lower resolution inference allows significant speedups in the processing, reaching 372 FPS at the highest scale -6x faster than SSCNet and 300x faster than Input SSCNet-full [36] LMSCNet ( [36], our LMSCNet provides smoother semantics labels and is capable of retrieving finer details. This is evident when looking at the cars (rows 7-8) or the trees (rows 5-6).
TS3D+DNet+SATNet -. Because semantic completion is an application of high interest for mobile robotics, like autonomous driving, our lighter architecture is beneficial for embedded GPUs and enables coarse scene analysis at high speed.

Ablation studies
To study the benefit of our design choices, we conduct a series of ablation studies on SemanticKITTI validation set. This is done by modifying important blocks of our architecture and evaluating its performance. Input SSCNet-full [36] LMSCNet (ours) Figure 5: Inference results on nuScenes [3]    Influence of input resolution. We evaluate our robustness, by retrieving the original 64-layers KITTI scans used in SemanticKITTI and simulating 8/16/32 layers LiDARs with layers subsampling 3 , as in [20]. Fig. 6 shows quantitative and qualitative performance using simulated and original LiDAR. As expected, lower layers input deteriorate the performance, especially in areas far from the sensor location, but our network still performs reasonably well on semantics (mIoU) and completion (IoU). This is visible in the middle image, as 8 layers input (2.10% density) is sufficient to retrieve the general outline of the scene. Deconv versus Upsampling. As we aimed to preserve a lightweight architecture, we tried to remove the parametersgreedy deconv layers from our network (cf. Sec. 2), replacing them with up-sampling layers. From Tab. 4, performance without deconv introduces a 1.43% and 1.14% performance drop for completion and semantic completion respectively, with only 3% less parameters.
Dilated convolutions. We evaluate the benefit of dilated convolutions in the decoder by ablating ASPP blocks from the segmentation head (see Fig. 2). Tab. 4 indicates that mIoU drops by 0.41% without ASPP. We conjecture that the boost of ASSP results come from the increasing receptive fields of the inner dilated convolutions, providing richer features.
Multiscale UNet decoder. As illustrated in Fig. 7a, unlike vanilla UNet decoder we concatenate the features at the end of each decoder level to all other levels. This is intended to aggregate multiscale features and should intuitively help considering coarser semantic features for fine resolutions.
We assess the benefit of our multiscale UNet by evaluating Vanilla UNet in the last row of Tab. 4, which shows that our proposal boosts completion by 0.68% and semantic completion by 0.56%.

Conclusion
We proposed a novel method, coined LMSCNet, for 3D Semantic Scene Completion, which benefits from mixing 2D/3D convolutions to preserve lightweight architecture, while enabling inference at multiple scales. On the challenging SemanticKITTI benchmark, we perform on par with other methods on semantic completion with a much lighter architecture and at faster inference speed. For completion, we outperform the state of the art. Also of interest for mobile robotics, our proposal is robust to much lower input density and our multiscale capacity enables scene completion for lower resolution at very high speed. Figure 9: Network speed (FPS) and performance (mIoU). FPS are shown in log-scale. Notice that our method keeps good performance and fast inference time, this is specially noticeable for our coarse scale versions LMSCNet (1:x). seen in main article Tab. 3. Considering this, our network keeps the best speed-performance trade-off. The interest of the coarser scale inferences of our method can be highlighted by the considerably lower inference times and high performances.

B. Qualitative results
In Fig. 8 further qualitative results of our method in both SemanticKITTI [1] and nuScenes [3] datasets are provided. Notice our network performs smoother and less noisy reconstructions. Even though the ground-truth in Se-manticKITTI accumulates scans of dynamic objects as seen in rows 3-4, our network reconstructs the vehicles correctly. This can be explained by the abundance of parked vehicles in the dataset. Performance in nuScenes can be seen in rows 5 to 8. It can be observed again the smoothness of the reconstruction when compared to SSCNet-full, with less noisy objects in the middle of the road. Notice that the network has been trained on SemanticKITTI, which explains the high vegetation predictions in nuScenes. We refer the reader to the supplementary video for more qualitative insights.