{'fraction_non_alphanumeric': 0.03002528445006321, 'fraction_numerical': 0.01828607549214376, 'mean_word_length': 4.824086247699185, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Point Cloud Segmentation Using Sparse Temporal Local Attention

Point clouds are a key modality used for perception in autonomous vehicles, providing the means for a robust geometric understanding of the surrounding environment. However despite the sensor outputs from autonomous vehicles being naturally temporal in nature, there is still limited exploration of exploiting point cloud sequences for 3D seman-tic segmentation. In this paper we propose a novel Sparse Temporal Local Attention (STELA) module which aggregates intermediate features from a local neighbourhood in previous point cloud frames to provide a rich temporal context to the decoder. Using the sparse local neighbourhood enables our approach to gather features more flexibly than those which directly match point features, and more efficiently than those which perform expensive global attention over the whole point cloud frame. We achieve a competitive mIoU of 64.3% on the SemanticKitti dataset, and demonstrate significant improvement over the single-frame baseline in our ablation studies.


Introduction
Semantic scene understanding is an essential task for the operation of autonomous vehicles. A fine-grained understanding of the environment allows an autonomous vehicle to distinguish between drivable and non-drivable surfaces, identify moving objects in a scene, and leverage additional context for improved place recognition [Vidanapathirana et al., 2021] or global data association for Simultaneous Localization and Mapping (SLAM) [Park et al., 2021]. Conversely, a poor understanding of the vehicle's surroundings can negatively impact its ability to make informed decisions and can lead to fatal and other unwanted accidents.
3D point clouds collected by either LiDAR or depth cameras are a common perception modality for many autonomous vehicles, providing better geometric understanding of the surrounding environment compared to a 2D camera. With the release of several large-scale annotated datasets in recent years [Behley et al., 2019;Caesar et al., 2020], the task of semantic segmentation of point clouds has gained in popularity. However, despite a number of successful recent approaches exploiting sequential data from 2D video streams for improved segmentation performance [Hu et al., 2020a;Li et al., 2018;Paul et al., 2020;Zhu et al., 2019;Jain et al., 2019], there has been limited exploration into leveraging temporal priors for point cloud segmentation. Existing approaches either calculate strict correspondences between point features across frames [Cao et al., 2020] or perform global attention [Shi et al., 2020] between whole point clouds. In the case of the former, a breakdown of nearest-point matching due to displacement between adjacent point clouds can result in the mis-matching of features across frames. In the case of the latter, the computational cost of performing global attention scales rapidly with the size of the point cloud and also risks introducing false correlations between superficially similar but distant regions of the scene [Paul et al., 2021].
We believe that effective leverage of point cloud sequences should gather features from a local region of the point cloud, allowing for more flexible aggregation of past information than strict correspondences while avoiding the pitfalls of global attention computations. To accomplish this we introduce a Sparse Temporal Local Attention (STELA) module as seen in Figure 2. This module decomposes the sparse 3D voxel features in the network backbone into pseudo-point clouds as shown in Figure 3, composed of the non-empty voxel features and their corresponding grid co-ordinates. The grid co-ordinates are used to find the spatially local neighbourhood between the past and present point cloud frames, over which we use cross-attention to aggregate salient features to provide useful temporal context to the decoder. By constraining our cross-attention to local neighbourhoods, we focus the search for correlated past features to the most relevant region of the point cloud while also significantly reducing the computational cost involved. In summary our contributions are: • We introduce the Sparse Temporal Local Attention (STELA) module which aggregates local features from previous point cloud frames to improve segmentation arXiv:2112.00289v2 [cs.CV] 2 Dec 2021 performance.
• We investigate the impact of the local neighbourhood size and sequence length on downstream performance when aggregating features from previous point cloud frames.
• We achieve competitive results on the SemanticKitti dataset, and demonstrate a significant boost in performance over the single-frame baseline.
2 Related Works

Semantic Segmentation in Static Point Clouds
Previous work in semantic segmentation for static point clouds can broadly be classified into three categories: Voxelisation, 2D projection and direct point-based methods. Voxelisation based approaches [Riegler et al., 2017;Tchapmi et al., 2017;Wang and Lu, 2019;Huang and You, 2016;Zhu et al., 2021;Graham et al., 2018] partition the point cloud into a 3D tensor, over which 3D convolutions can be used to extract and classify point features. However, the large size and sparsity of these voxelisations can present issues with memory consumption as the number of voxels increases. To resolve this, cylindrical partitioning [Zhu et al., 2021] and sparse convolutions [Graham et al., 2018] have been used to reduce the sparsity of the partition and restrict operations to the active regions of the tensor. 2D projection based methods project the 3D point cloud onto a 2D plane by using the spherical projection Wu et al., 2018;Wu et al., 2019;Cortinhal et al., 2020], bird's-eye projection [Zhang et al., 2020c;Aksoy et al., 2019] or a fused approach [Alnaggar et al., 2021;. This family of approaches tends to reach higher segmentation speed due to the efficient use of 2D convolutions, but suffers from the loss of geometric information in the point cloud during projection from 3D to 2D.
Direct point-based methods process the point cloud directly instead of projecting to a tensor representation. The majority of approaches in this category are based on the PointNet architecture [Qi et al., 2017b;Qi et al., 2017a] which extracts point-wise features using a multi-layer perceptron (MLP) for segmentation. Recent approaches have focused on extracting the local relationships between points through improved grouping [Engelmann et al., 2018] or crafting convolutional operations to take place over the point cloud [Groh et al., 2018;Tatarchenko et al., 2018;.

Semantic Segmentation in Point Cloud Sequences
Despite significant recent gains made in video semantic segmentation by incorporating sequential data [Hu et al., 2020a;Li et al., 2018;Paul et al., 2020;Zhu et al., 2019;Jain et al., 2019], few approaches for point cloud segmentation have taken advantage of this temporal signal. MeteorNet [Liu et al., 2019] and ASAP-Net [Cao et al., 2020] are point-based methods which use 3D flow and nearest-centre matching respectively to identify point features from past point cloud frames to be fused into the present timestep. SpSequenceNet [Shi et al., 2020] is a voxelisation method which performs global attention to fuse highly correlated features between point cloud frames. However, MeteorNet and ASAP-Net's reliance on flow and centre matching to calculate strict correspondence leads to issues when these priors fail, while SpSe-quenceNet's use of global attention is computationally expensive and introduces the possibility of finding false correlations between similar instances or classes present in different areas of the scene [Paul et al., 2021].

Temporal Cross-Attention
Cross-attention is a variant of the attention mechanisms which are used for tasks such as sensor fusion [Mohla et al., 2020;Pande and Banerjee, 2021;Zhang et al., 2020b] and image captioning [Wei et al., 2020;Lee et al., 2018] where it is important to find correlations between modalities which provide different views of the same target. We define temporal cross-attention as an emerging family of methods [Shi et al., 2020;Cao et al., 2020;Paul et al., 2021;Bertasius et al., 2021] in segmentation and classification tasks which leverage the attention mechanism by treating subsequent point cloud frames of the input as different 'views' of a scene. One of the advantages of employing temporal crossattention is the ability to focus the search for correlations to a local neighbourhood, as the temporally smooth nature of subsequent point cloud frames suggests that the most salient regions of the input for finding strong correlations will occupy a similar spatial location between point cloud frames. This prior allows for the construction of a local neighbourhood for the attention mechanism which avoids the pitfalls of mis-matching features when strict correlation between features across point cloud frames is used [Cao et al., 2020], significantly reducing the computational cost and chance of false correlations compared to global attention [Shi et al., 2020].

Methodology
In this section we describe the learning framework, details of the architecture and loss function, and other modules used to implement our proposed sparse attention mechanism for point cloud sequences.

Framework Overview
Strong correlations exist in the semantic properties of a scene between frames, so it is natural to exploit this temporal consistency instead of processing each point cloud frame independently. However, point clouds for outdoor scenes contain several properties which impose challenges to drawing on these temporal correlations: they are sparse, have variable input size, and points relating to the same object in the world may be occluded in subsequent frames due to changes in the sensor's viewpoint. In order to address these issues we propose a novel Sparse Temporal Local Attention module (STELA), as depicted in Figure 2. Section 3.2 outlines the projection of the point cloud sequence into sparse cylindrical partitions. Section 3.3 outlines the distinction between performing local and global attention for feature correlation, and Section 3.4 describes how we find the sparse local neighbourhood between point cloud frames. Finally Sections 3.5 and 3.6 detail how we use attention to find correlations over these local neighbourhoods and produce an aggregated embedding of features from previous point cloud frames, and fuse these features with the present to provide an input for the decoder.

Partitioning the Point Cloud Cylindrical Representation
Outdoor point clouds vary significantly in density based on distance to the LiDAR sensor. In order to achieve a lower proportion of empty voxels in the partitioned point cloud, we employ a cylindrical projection which transforms our input points from co-ordinate system (x, y, z) to (p, θ, z) where p, θ represent the range (distance to the origin along the x-y plane) and azimuth (angle from x-axis to y-axis) of a point respectively. This style of partition is suited to outdoor scenes, since the increasing grid size compensates for the growth in sparsity as the range from the sensor increases. This leads to a more balanced distribution of points in the input partition and a more even proportion of non-empty voxels [Zhu et al., 2021]. We then use a combination of the original x, y, z co-ordinates, the newly found p, θ co-ordinates and the LiDAR intensity as the input for a four-layer MLP to generate the features from each point, and assign them to the partition based on their cylindrical co-ordinates. After these steps, we unroll the cylinder to get the 3D feature embedding F ∈ R D×H×W ×L , where D denotes the feature dimension and H, W, L the range, azimuth and height respectively.

Sparse Feature Maps as Pseudo-Point Clouds
Inspired by sparse convolutional methods, given an input sparse 3D feature map F ∈ R D×H×W ×L we decompose the useful information in the embedding into two tensors V ∈ R N ×D and I ∈ R N ×3 . Here V and I represent the features and grid indexes of the N non-empty voxels in the embedding as illustrated in Figure 3. We use these features and grid indexes to calculate our keys and local neighbourhood for the STELA module as described below, and convert them back to sparse 3D partitions between modules for 3D convolution.

Local vs. Global Attention
When aggregating past features, a global matching can be problematic for several reasons. Large 3D feature maps can lead to unreasonable memory consumption and degradation of inference time, scaling rapidly as the size of the partition increases. In addition, matching features at distant locations has the potential to introduce false correlations between similar instances or classes in a scene [Paul et al., 2021]. For these reasons we spatially align the point clouds from previous frames with the present using the local sensor odometry  and constrain our attention mechanism for each voxel to a spatially local neighbourhood.

Sparse Local Neighbourhood
Unlike methods applying local attention in the video domain [Paul et al., 2021], given the sparse nature of the 3D partitions used to represent outdoor point cloud scenes, simply using the spatially adjacent voxels for our local neighbourhood would result in redundant calculations over a large number of empty voxels during training and inference. In this section, we introduce a novel method of finding the local neighbourhood given a sparse input in order to avoid these redundant correlations. Let f t = {V t , I t } and f t−n = {V t−n , I t−n } represent the decomposed features for the point cloud embeddings at times t and t − n. To find our local neighbourhood L t−n (i) for a given non-empty voxel i ∈ {1, . . . , N } in f t , we find the k spatially closest non-empty voxels in f t−n using K-Nearest Neighbours, with L t−n (i) representing the set of non-empty voxels in the local neighbourhood for past point cloud frame t − n, corresponding to a non-empty voxel in the current frame t. Similarly to f t−n , each voxel in L t−n has an associated feature and grid index such that L t−n = V L t−n , I L t−n , where V L t−n ∈ R k×D and I L t−n ∈ R k×3 .

Keys Matching and Memory Reading
In order to find the correlations between past and present point cloud frames, we first pass our voxel features through an adapter function γ (·) in order to create a lower dimensional key embedding with feature dimensionality D K . For this adapter function we have chosen a two-layer MLP. Correlations can then be found by calculating the scaled dot product, where C t−n (i) ∈ R N ×k represents the correlation scores between the non-empty voxels in the point cloud frame at time t and its local neighbourhood in the frame at time t − n. We can then convert this to a mask P t−n for the past features by taking the SoftMax over the correlations for all point cloud frames, where j ∈ {1, . . . , k} represents the range over the local neighbourhood. We can now aggregate past features to create our memory tensor V M as follows, This memory tensor represents a weighted sum of all past features over the n past point cloud frames and local neighbourhood k. By gathering these features we are able to provide key temporal context to the decoder, improving our downstream performance.

Memory Fusion
When fusing the gathered past and present features, ideally the network should be able to identify when to trust the past features and when circumstances such as objects entering the scene, occlusions, or irregular motions may impact the trustworthiness of the gathered features. To do this we adapt the fusion module introduced by [Paul et al., 2021] to the point cloud domain, concatenating the inputs to create a sigmoidgating function to weigh the relative inputs of the past and present features. Formally, this function can be expressed as where σ represents the sigmoid activation function and G t , G M are the learned linear layers for masking the inputs of the present and memory features respectively.

Experimental Results
In this section first we introduce the datasets used in our experiments and the details surrounding our choice of network architecture and hyper-parameters. Next, we report results on ablation studies surrounding the impact of using frame alignment, the size of our local neighbourhood and the number of past frames used. Finally, we compare our network trained on the full SemanticKitti dataset to state-of-the-art results.

Dataset and Metric
SemanticKitti SemanticKitti [Behley et al., 2019] is a large-scale driving dataset for point cloud segmentation, containing labels for both semantic and panoptic segmentation. The dataset consists of 22 sequences, splitting sequences 00 to 10 as training set (where sequence 08 is used as the validation set), and sequences 11 to 21 as test set leading to a total of 43,351 frames overall. After merging moving and non-moving labels there are 19 foreground classes used for training and testing, with a small number of ignore labels.

Tiny SemanticKitti
Tiny SemanticKitti is a subset of the SemanticKitti dataset we introduce in this paper, with a train set consisting of sequence 0 of the SemanticKitti dataset and a validation set consisting of every 10th frame in sequence 8. We use Tiny SemanticKitti to perform our ablation studies, but report the final results in Table 1 by training on the full SemanticKitti train set and testing with the full SemanticKitti test set.

Evaluation Metric
To evaluate our proposed method we use the mean intersection over union metric (mIoU). This metric can be formulated Where TP c , FP c , FN c represent the true positives, false positives and false negative predictions for class c.

Implementation Details
To show the importance of temporal information, we used a basic encoder-decoder architecture along with our STELA module as seen in Figure 1. The output from each downsample block is updated using a STELA module before being passed to its corresponding upsample block through a skip connection. This architecture is adapted from [Zhu et al., 2021], using cylindrical voxel co-ordinates and asymmetrical convolution in the downsample and upsample blocks. However, we do not adopt the additional point-based feature branch used in their approach. We use an input voxel size of 240 × 180 × 16 for the ablation studies and 480 × 360 × 32 for our results compared to the state-of-the-art. We train with a batch size of 4 on 4 Tesla-P100 GPUs.
Our training takes place in three stages. Firstly, we train a model without the STELA module to perform single-frame  [Tang et al., 2020] 66 [Zhu et al., 2021] 67 segmentation. Next, we insert the STELA module into each skip layer of the network and freeze the encoder and decoder in order to initialize the weights of the module. Finally, we unfreeze the network and train end-to-end for 40 epochs or until the model reaches convergence. We use a fixed learning rate of 1e −3 for the first two stages of training, and a learning rate of 1e −5 decaying on plateau for the final step. For our loss function, we follow existing methods [Cortinhal et al., 2020;Zhu et al., 2021] and use a combination of the weighted cross-entropy and lovasz-softmax [Berman et al., 2018] losses to maximise the accuracy and IoU score of the network output respectively.

Ablation Studies Aligning Inputs
Method Aligned mIoU ∆% Baseline -52.57 -STELA 52.87 0.3% STELA 53.97 1.4% Table 2: Impact of aligning point clouds with current frame on segmentation performance on Tiny SemanticKitti. We use a local neighbourhood size and sequencehttps://www.overleaf.com/project/601ccddf620035b0019fbed0 length of k = 16 and n = 2 for this ablation. Table 2 shows the result of aligning the point clouds into the sensor reference frame at time t. Baseline refers to the performance of the network architecture without the STELA module, taking a single frame as an input. Aligning the point clouds results in a 1.1% boost in performance from the STELA module, though a small improvement is still observed when alignment does not take place. Table 3 shows the effect of the size of the local neighbourhood, measured by the number of k nearest non-empty neighbours. We can see that increasing the size of the local neighbourhood causes incremental improvements in the   Table 4: Impact of past frames used on segmentation performance on Tiny SemanticKitti. We use a local neighbourhood size of k = 16 for this ablation. Table 4 shows the effect of changing the number of frames used in the sequence input to the STELA module. We achieve our strongest result using a sequence length containing two past point cloud frames (total sequence length three), after which performance begins to decline. This is likely since as the time between frames increases, the increasing differences between present and past diminishes the usefulness of the prior features. However, our result demonstrates that there is benefit in searching for correlations over more than a single past frame. Table 1 compares our result to the state-of-the-art on the test split of the SemanticKITTI dataset. We achieve competitive results with the state-of-art, outperforming to the best of our knowledge all but two other methods on the SemanticKitti test set. We believe this strong overall result combined with our demonstrated improvements on the Tiny SemanticKitti validation set confirm the benefits of our approach over the single-frame baseline.

Conclusion
In this paper we propose the novel attention module STELA to aggregate sequential information from temporally adjacent point clouds. By restricting our attention mechanism to the local spatiotemporal neighbourhood, we improve performance by focusing on the most relevant past features while also reducing the computational expense of finding correlations. For future work we wish to investigate improving our attention mechanism by incorporating stronger geometric priors such as a positional embedding into the keys, and integrating learned odometry into the pipeline in order to reduce reliance on external priors for frame alignment.