{'fraction_non_alphanumeric': 0.032254439199820184, 'fraction_numerical': 0.0181314152993182, 'mean_word_length': 4.98542600896861, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Point Transformer

Self-attention networks have revolutionized natural language processing and are making impressive strides in image analysis tasks such as image classification and object detection. Inspired by this success, we investigate the application of self-attention networks to 3D point cloud processing. We design self-attention layers for point clouds and use these to construct self-attention networks for tasks such as semantic scene segmentation, object part segmentation, and object classification. Our Point Transformer design improves upon prior work across domains and tasks. For example, on the challenging S3DIS dataset for large-scale semantic scene segmentation, the Point Transformer attains an mIoU of 70.4% on Area 5, outperforming the strongest prior model by 3.3 absolute percentage points and crossing the 70% mIoU threshold for the first time.


Introduction
3D data arises in many application areas such as autonomous driving, augmented reality, and robotics. Unlike images, which are arranged on regular pixel grids, 3D point clouds are sets embedded in continuous space. This makes 3D point clouds structurally different from images and precludes immediate application of deep network designs that have become standard in computer vision, such as networks based on the discrete convolution operator.
A variety of approaches to deep learning on 3D point clouds have arisen in response to this challenge. Some voxelize the 3D space to enable the application of 3D discrete convolutions [23,32]. This induces massive computational and memory costs and underutilizes the sparsity of point sets in 3D. Sparse convolutional networks relieve these limitations by operating only on voxels that are not empty [9,3]. Other designs operate directly on points and propagate information via pooling operators [25,27] or continuous convolutions [42,37]. Another family of approaches connect the point set into a graph for message passing [44,19].
In this work, we develop an approach to deep learning on point clouds that is inspired by the success of transformers in natural language processing [39,45,5,4,51] and image analysis [10,28,54]. The transformer family of models is particularly appropriate for point cloud processing because the self-attention operator, which is at the core of transformer networks, is in essence a set operator: it is invariant to permutation and cardinality of the input elements. The application of self-attention to 3D point clouds is therefore quite natural, since point clouds are essentially sets embedded in 3D space. We flesh out this intuition and develop a self-attention layer for 3D point cloud processing. Based on this layer, we construct Point Transformer networks for a variety of 3D understanding tasks. We investigate the form of the selfattention operator, the application of self-attention to local neighborhoods around each point, and the encoding of positional information in the network. The resulting networks are based purely on self-attention and pointwise operations.
We show that Point Transformers are remarkably effective in 3D deep learning tasks, both at the level of detailed object analysis and large-scale parsing of massive scenes. In particular, Point Transformers set the new state of the art on large-scale semantic segmentation on the S3DIS dataset (70.4% mIoU on Area 5), shape classification on Model-Net40 (93.7% overall accuracy), and object part segmenta-tion on ShapeNetPart (86.6% instance mIoU). Our full implementation and trained models will be released upon acceptance. In summary, our main contributions include the following.
• We design a highly expressive Point Transformer layer for point cloud processing. The layer is invariant to permutation and cardinality and is thus inherently suited to point cloud processing.
• Based on the Point Transformer layer, we construct high-performing Point Transformer networks for classification and dense prediction on point clouds. These networks can serve as general backbones for 3D scene understanding.
• We report extensive experiments over multiple domains and datasets. We conduct controlled studies to examine specific choices in the Point Transformer design and set the new state of the art on multiple highly competitive benchmarks, outperforming long lines of prior work.

Related Work
For 2D image understanding, pixels are placed in regular grids and can be processed with classical convolution. In contrast, 3D point clouds are unordered and scattered in 3D space: they are essentially sets. Learning-based approaches to processing 3D point clouds can be classified into the following types: projection-based, voxel-based, and point-based networks. Projection-based networks. For processing irregular inputs like point clouds, an intuitive way is to transform irregular representations to regular ones. Considering the success of 2D CNNs, some approaches [34,18,2,14,16] adopt multi-view projection, where 3D point clouds are projected into various image planes. Then 2D CNNs are used to extract feature representations in these image planes, followed by multi-view feature fusion to form the final output representations. In a related approach, TangentConv [35] projects local surface geometry onto a tangent plane at every point, forming tangent images that can be processed by 2D convolution. However, this approach heavily relies on tangent estimation. In projection-based frameworks, the geometric information inside point clouds is collapsed during the projection stage. These approaches may also underutilize the sparsity of point clouds when forming dense pixel grids on projection planes. The choice of projection planes may heavily influence recognition performance and occlusion in 3D may impede accuracy. Voxel-based networks. An alternative approach to transforming irregular point clouds to regular representations is 3D voxelization [23,32], followed by convolutions in 3D. When applied naively, this strategy can incur massive com-putation and memory costs due to the cubic growth in the number of voxels as a function of resolution. The solution is to take advantage of sparsity, as most voxels are usually unoccupied. For example, OctNet [29] uses unbalanced octrees with hierarchical partitions. Approaches based on sparse convolutions, where the convolution kernel is only evaluated at occupied voxels, can further reduce computation and memory requirements [9,3]. These methods have demonstrated good accuracy but may still lose geometric detail due to quantization onto the voxel grid. Point-based networks. Rather than projecting or quantizing irregular point clouds onto regular grids in 2D or 3D, researchers have designed deep network structures that ingest point clouds directly, as sets embedded in continuous space. PointNet [25] utilizes permutation-invariant operators such as pointwise MLPs and pooling layers to aggregate features across a set. PointNet++ [27] applies these ideas within a hierarchical spatial structure to increase sensitivity to local geometric layout. Such models can benefit from efficient sampling of the point set, and a variety of sampling strategies have been developed [27,7,46,50,11].
A number of approaches connect the point set into a graph and conduct message passing on this graph. DGCNN [44] performs graph convolutions on kNN graphs. PointWeb [55] densely connects local neightborhoods. ECC [31] uses dynamic edge-conditioned filters where convolution kernels are generated based on edges inside point clouds. SPG [15] operates on a superpoint graph that represents contextual relationships. KCNet [30] utilizes kernel correlation and graph pooling. Wang et al. [40] investigate the local spectral graph convolution. GACNet [41] employs graph attention convolution and HPEIN [13] builds a hierarchical point-edge interaction architecture. DeepGCNs [19] explore the advantages of depth in graph convolutional networks for 3D scene understanding.
A number of methods are based on continuous convolutions that apply directly to the 3D point set, with no quantization. PCCN [42] represents convolutional kernels as MLPs. SpiderCNN [49] defines kernel weights as a family of polynomial functions. Spherical CNN [8] designs spherical convolution to address the problem of 3D rotation equivariance. PointConv [46] and KPConv [37] construct convolution weights based on the input coordinates. InterpCNN [22] utilizes coordinates to interpolate pointwise kernel weights. PointCNN [20] proposes to reorder the input unordered point clouds with special operators. Ummenhofer et al. [38] apply continuous convolutions to learn particle-based fluid dynamics. Transformer and self-attention. Transformer and selfattention models have revolutionized machine translation and natural language processing [39,45,5,4,51]. This has inspired the development of self-attention networks for 2D image recognition [10,28,54,6]. Hu et al. [10] and Ramachandran et al. [28] apply scalar dot-product selfattention within local image patches. Zhao et al. [54] develop a family of vector self-attention operators. Dosovitskiy et al. [6] treat images as sequences of patches.
Our work is inspired by the findings that transformers and self-attention networks can match or even outperform convolutional networks on sequences and 2D images. Self-attention is of particular interest in our setting because it is intrinsically a set operator: positional information is provided as attributes of elements that are processed as a set [39,54]. Since 3D point clouds are essentially sets of points with positional attributes, the self-attention mechanism seems particularly suitable to this type of data. We thus develop a Point Transformer layer that applies selfattention to 3D point clouds.
There are a number of previous works [48,21,50,17] that utilize attention for point cloud analysis. They apply global attention on the whole point cloud, which introduces heavy computation and renders these approaches inapplicable to large-scale 3D scene understanding. They also utilize scalar dot-product attention, where different channels share the same aggregation weights. In contrast, we apply selfattention locally, which enables scalability to large scenes with millions of points, and we utilize vector attention, which we show to be important for achieving high accuracy. We also demonstrate the importance of appropriate position encoding in large-scale point cloud understanding, in contrast to prior approaches that omitted position information. Overall, we show that appropriately designed self-attention networks can scale to large and complex 3D scenes, and substantially advance the state of the art in large-scale point cloud understanding.

Point Transformer
We begin by briefly revisiting the general formulation of transformers and self-attention operators. Then we present the point transformer layer for 3D point cloud processing. Lastly, we present our network architecture for 3D scene understanding.
Let X = {x i } i be a set of feature vectors. The standard scalar dot-product attention layer can be represented as follows: where y i is the output feature. φ, ψ, and α are pointwise feature transformations, such as linear projections or MLPs. δ is a position encoding function and ρ is a normalization function such as softmax. The scalar attention layer computes the scalar product between features transformed by φ and ψ and uses the output as an attention weight for aggregating features transformed by α.
In vector attention, the computation of attention weights is different. In particular, attention weights are vectors that can modulate individual feature channels: where β is a relation function (e.g., subtraction) and γ is a mapping function (e.g., an MLP) that produces attention vectors for feature aggregation.
Both scalar and vector self-attention are set operators. The set can be a collection of feature vectors that represent the entire signal (e.g., sentence or image) [39,6] or a collection of feature vectors from a local patch within the signal (e.g., an image patch) [10,28,54].

Point Transformer Layer
Self-attention is a natural fit for point clouds because point clouds are essentially sets embedded irregularly in a metric space. Our point transformer layer is based on vector self-attention. We use the subtraction relation and add a position encoding δ to both the attention vector γ and the transformed features α: Here the subset X (i) ⊆ X is a set of points in a local neighborhood (specifically, k nearest neighbors) of x i . Thus we adopt the practice of recent self-attention networks for image analysis in applying self-attention locally, within a local neighborhood around each datapoint [10,28,54]. The mapping function γ is an MLP with two linear layers and one ReLU nonlinearity. The point transformer layer is illustrated in Figure 2.

Position Encoding
Position encoding plays an important role in selfattention, allowing the operator to adapt to local structure in the data [39]. Standard position encoding schemes for sequences and image grids are crafted manually, for example based on sine and cosine functions or normalized range values [39,54]. In 3D point cloud processing, the 3D point coordinates themselves are a natural candidate for position encoding. We go beyond this by introducing trainable, parameterized position encoding. Our position encoding function δ is defined as follows: Here p i and p j are the 3D point coordinates for points i and j. The encoding function θ is an MLP with two linear layers and one ReLU nonlinearity. Notably, we found that position encoding is important for both the attention generation branch and the feature transformation branch. Thus Eq. 3 adds the trainable position encoding in both branches. The position encoding θ is trained end-to-end with the other subnetworks.

Point Transformer Block
We construct a residual point transformer block with the point transformer layer at its core, as shown in Figure 4(a). The transformer block integrates the self-attention layer, linear projections that can reduce dimensionality and accelerate processing, and a residual connection. The input is a set of feature vectors x with associated 3D coordinates p. The point transformer block facilitates information exchange between these localized feature vectors, producing new feature vectors for all data points as its output. The information aggregation adapts both to the content of the feature vectors and their layout in 3D.

Network Architecture
We construct complete 3D point cloud understanding networks based on the point transformer block. Note that the point transformer is the primary feature aggregation operator throughout the network. We do not use convolutions for preprocessing or auxiliary branches: the network is based entirely on point transformer layers, pointwise transformations, and pooling. The network architectures are visualized in Figure 3.
Backbone structure. The feature encoder in point transformer networks for semantic segmentation and classification has five stages that operate on progressively downsampled point sets. The downsampling rates for the stages are [1,4,4,4,4], thus the cardinality of the point set produced by each stage is [N, N/4, N/16, N/64, N/256], where N is the number of input points. Note that the number of stages and the downsampling rates can be varied depending on the application, for example to construct light-weight backbones for fast processing. Consecutive stages are connected by transition modules: transition down for feature encoding and transition up for feature decoding.

Transition down.
A key function of the transition down module is to reduce the cardinality of the point set as required, for example from N to N/4 in the transition from the first to the second stage. Denote the point set provided as input to the transition down module as P 1 and denote the output point set as P 2 . We perform farthest point sampling [27] in P 1 to identify a well-spread subset P 2 ⊂ P 1 with the requisite cardinality. To pool feature vectors from P 1 onto P 2 , we use a kNN graph on P 1 . (This is the same k as in Section 3.2. We use k = 16 throughout and report a controlled study of this hyperparameter in Section 4.4.) Each input feature goes through a linear transformation, followed by batch normalization and ReLU, followed by max pooling onto each point in P 2 from its k neighbors in P 1 . The transition down module is schematically illustrated in Figure 4(b).
Transition up. For dense prediction tasks such as semantic segmentation, we adopt a U-net design in which the encoder described above is coupled with a symmetric decoder [27,3]. Consecutive stages in the decoder are connected by transition up modules. Their primary function is to map features from the downsampled input point set P 2 onto its superset P 1 ⊃ P 2 . To this end, each input point feature is processed by a linear layer, followed by batch normalization and ReLU, and then the features are mapped onto the higher-resolution point set P 1 via trilinear interpolation. These interpolated features from the preceding decoder stage are summarized with the features from the corresponding encoder stage, provided via a skip connection. The structure of the transition up module is illustrated in Figure 4(c).
Output head. For semantic segmentation, the final decoder stage produces a feature vector for each point in the input point set. We apply an MLP to map this feature to the final logits. For classification, we perform global average pooling over the pointwise features to get a global feature vector for the whole point set. This global feature is passed through an MLP to get the global classification logits.

Experiments
We evaluate the effectiveness of the presented Point Transformer design on a number of domains and tasks. For 3D semantic segmentation, we use the challenging Stanford Large-Scale 3D Indoor Spaces (S3DIS) dataset [1]. For 3D shape classification, we use the widely adopted Mod-elNet40 dataset [47]. And for object part segmentation, we use ShapeNetPart [52]. Implementation details. We implement the Point Transformer in PyTorch [24]. We use the SGD optimizer with momentum and weight decay set to 0.9 and 0.0001, respectively. For semantic segmentation on S3DIS, we train for 40K iterations with initial learning rate 0.5, dropped by 10x at steps 24K and 32K. For 3D shape classification on Mod-elNet40 and 3D object part segmentation on ShapeNetPart, we train for 200 epochs. The initial learning rate is set to 0.05 and is dropped by 10x at epochs 120 and 160.

Semantic Segmentation
Data and metric. The S3DIS [1] dataset for semantic scene parsing consists of 271 rooms in six areas from three different buildings. Each point in the scan is assigned a semantic label from 13 categories (ceiling, floor, table, etc.). Following a common protocol [36,27], we evaluate the presented approach in two modes: (a) Area 5 is withheld during training and is used for testing, and (b) 6-fold cross-validation. For evaluation metrics, we use mean classwise intersection over union (mIoU), mean of classwise accuracy (mAcc), and overall pointwise accuracy (OA). Performance comparison. The results are presented in Tables 1 and 2. The Point Transformer outperforms all prior models according to all metrics in both evaluation modes. On Area 5, the Point Transformer attains mIoU/mAcc/OA of 70.4%/76.5%/90.8%, outperforming all prior work by multiple percentage points in each metric. The Point Transformer is the first model to pass the 70% mIoU bar, outperforming the prior state of the art by 3.3 absolute percentage points in mIoU. The Point Transformer outperforms MLPs-based frameworks such as PointNet [25], voxelbased architectures such as SegCloud [36], graph-based methods such as SPGraph [15], attention-based methods such as PAT [50], sparse convolutional networks such as MinkowskiNet [3], and continuous convolutional networks such as KPConv [37]. Point Transformer also substantially outperforms all prior models under 6-fold cross-validation. The mIoU in this mode is 73.5%, outperforming the prior state of the art (KPConv) by 2.9 absolute percentage points. The number of parameters in Point Transformer (4.9M) is much smaller than in current high-performing architectures such as KPConv (14.9M) and SparseConv (30.1M). Visualization. Figure 5 shows the Point Transformer's predictions. We can see that the predictions are very close to

Shape Classification
Data and metric. The ModelNet40 [47] dataset contains 12,311 CAD models with 40 object categories. They are split into 9,843 models for training and 2,468 for testing. We follow the data preparation procedure of Qi et al. [27] and uniformly sample the points from each CAD model together with the normal vectors from the object meshes.
For evaluation metrics, we use the mean accuracy within each category (mAcc) and the overall accuracy (OA) over all classes.
Performance comparison. The results are presented in Table 3. The Point Transformer sets the new state of the art in both metrics. The overall accuracy of Point Transformer on ModelNet40 is 93.7%. It outperforms strong graph-based models such as DGCNN [44], attention-based models such as A-SCN [48] and Point2Sequence [21], and strong pointbased models such as KPConv [37].
Visualization. To probe the representation learned by the Point Transformer, we conduct shape retrieval by retrieving nearest neighbors in the space of the output features produced by the Point Transformer. Some results are shown in Figure 6. The retrieved shapes are very similar to the query, and when they differ, they differ along aspects that we perceive as less semantically salient, such as legs of desks.

Object Part Segmentation
Data and metric. The ShapeNetPart dataset [52] is annotated for 3D object part segmentation. It consists of 16,880 models from 16 shape categories, with 14,006 3D models for training and 2,874 for testing. The number of parts for each category is between 2 and 6, with 50 different parts in total. We use the sampled point sets produced by Qi et al. [27] for a fair comparison with prior work. For evaluation metrics, we report category mIoU and instance mIoU.
Performance comparison. The results are presented in Ta Visualization. Object part segmentation results on a number of models are shown in Figure 7. The Point Transformer's part segmentation predictions are clean and close to the ground truth.

Ablation Study
We now conduct a number of controlled experiments that examine specific decisions in the Point Transformer design. These studies are performed on the semantic segmentation task on the S3DIS dataset, tested on Area 5.

Number of neighbors.
We first investigate the setting of the number of neighbors k, which is used in determining the local neighborhood around each point. The results are shown in Table 5. The best performance is achieved when k is set to 16. When the neighborhood is smaller (k = 4 or k = 8), the model may not have sufficient context for its predictions. When the neighborhood is larger (k = 32 or k = 64), each self-attention layer is provided with a large number of datapoints, many of which may be farther and less relevant. This may introduce excessive noise into the processing, lowering the model's accuracy.
Softmax regularization. We conduct an ablation study on the normalization function ρ in Eq. 3. The performance without softmax regularization on S3DIS Area5 is 66.5%/72.8%/89.3%, in terms of mIoU/mAcc/OA. It is much lower than the performance with softmax regularization (70.4%/76.5%90.8%). This suggests that the normalization is essential in this setting.
Position encoding. We now study the choice of the position encoding δ. The results are shown in Table 6. We can see that without position encoding, the performance drops significantly. With absolute position encoding, the performance is higher than without. Relative position encoding yields the highest performance. When relative position encoding is added only to the attention generation branch (first term in Eq. 3) or only to the feature transformation branch (second term in Eq. 3), the performance drops again, indicating that adding the relative position encoding to both branches is important.
Attention type. Finally, we investigate the type of selfattention used in the point transformer layer. The results are shown in Table 7. We examine four conditions. 'MLP' is a no-attention baseline that replaces the point transformer layer in the point transformer block with a pointwise MLP. 'MLP+pooling' is a more advanced no-attention baseline that replaces the point transformer layer with a pointwise MLP followed by max pooling within each kNN neighborhood: this performs feature transformation at each point and enables each point to exchange information with its local neighborhood, but does not leverage attention mechanisms. 'scalar attention' replaces the vector attention used in Eq. 3 by scalar attention, as in Eq. 1 and in the original transformer design [39]. 'vector attention' is the formulation we use, presented in Eq. 3. We can see that scalar attention is more expressive than the no-attention baselines, but is in turn outperformed by vector attention. The performance gap between vector and scalar attention is significant: 70.4% vs. 64.6%, an improvement of 5.8 absolute percentage points. Vector attention is more expressive since it supports adaptive modulation of individual feature channels, not just whole feature vectors. This expressivity appears to be very beneficial in 3D data processing.

Conclusion
Transformers have revolutionized natural language processing and are making impressive gains in 2D image analysis. Inspired by this progress, we have developed a transformer architecture for 3D point clouds. Transformers are perhaps an even more natural fit for point cloud processing than they are for language or image processing, because point clouds are essentially sets embedded in a metric space, and the self-attention operator at the core of trans-former networks is fundamentally a set operator. We have shown that beyond this conceptual compatibility, transformers are remarkably effective in point cloud processing, outperforming state-of-the-art designs from a variety of families: graph-based models, sparse convolutional networks, continuous convolutional networks, and others. We hope that our work will inspire further investigation of the properties of point transformers, the development of new operators and network designs, and the application of transformers to other tasks, such as 3D object detection.