{'fraction_non_alphanumeric': 0.03235874604121724, 'fraction_numerical': 0.011474732638729517, 'mean_word_length': 4.969315068493151, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VNT-Net: Rotational Invariant Vector Neuron Transformers

Learning 3D point sets with rotational invariance is an important and challenging problem in machine learning. Through rotational invariant architectures, 3D point cloud neural networks are relieved from requiring a canonical global pose and from exhaustive data augmentation with all possible rotations. In this work, we introduce a rotational invariant neural network by combining recently introduced vector neurons with self-attention layers to build a point cloud vector neuron transformer network (VNT-Net). Vector neurons are known for their simplicity and versatility in representing SO(3) actions and are thereby incorporated in common neural operations. Similarly, Transformer architectures have gained popularity and recently were shown successful for images by applying directly on sequences of image patches and achieving superior performance and convergence. In order to benefit from both worlds, we combine the two structures by mainly showing how to adapt the multi-headed attention layers to comply with vector neurons operations. Through this adaptation attention layers become SO(3) and the overall network becomes rotational invariant. Experiments demonstrate that our network efficiently handles 3D point cloud objects in arbitrary poses. We also show that our network achieves higher accuracy when compared to related state-of-the-art methods and requires less training due to a smaller number of hyperparameters in common classification and segmentation tasks.


Introduction
Unlike 2D images, point clouds lack an underlying regular structure, making it challenging to design neural networks to process them. Inspired by the progress of convolutional neural networks (CNNs) in the field of image processing, many recent point cloud networks aim at defining convolution operators that can handle unordered point sets and aggregate local features by incorporating orderinvariant and/or order-equivariant layers such as the multi-layer perceptron (MLP) of Qi et al. [25] and others [21,2,30].
While order invariance has been widely explored for point cloud neural networks, rotational invariance has been just starting to gain popularity. Nonetheless, it is crucial that point cloud neural networks are relieved from requiring all objects to be aligned in one canonical pose and from exhaustive data augmentation with all possible rotations.
For this purpose, there is a need for network layers that are equivariant to both order and SO (3) symmetries. Recently, two approaches have been introduced to tackle this: Tensor Field Networks [31] and SE(3)-Transformers [14]. While guaranteeing equivariance, both frameworks involve an intricate formulation and are hard to incorporate into existing pipelines as they are restricted to convolutions and rely on relative positions of adjacent points. To address these issues, a simple, lightweight framework to build SO(3) equivariant and invariant pointcloud networks was introduced based on Vector Neurons [8]. Their method defines a latent matrix representations which supports direct mapping of rotations from the input pointcloud to intermediate layers. 3D generalization of classical activation functions enables prediction of activation directions which in turn guarantee equivariant layers.
Inspired by the simplicity and efficiency of vector neurons, we take another step further in this path and show how to incorporate them into self attention layers (Transformers). Transformer networks have demonstrated promising results in natural language processing [32], images [10] and point cloud [15] tasks. In general, these models are superior in quality while being more parallelizable and requiring significantly less time to train.
In this work, we incorporate vector neurons into self attention layers to build a point cloud transformer network that is rotational invariant (see Figure 1). The core idea is that self attention layers typically perform on scalars. Instead, we adapt self attention layers to handle vector neural representation which enable SO(3) rotational equivariance. Thus, our vector neurons transformers network (VNT-Net) is rotational invariant. Furthermore it is both simple and achieves high quality results with less training due to a smaller number of hyper parameters.
Our method makes the following technical contributions: • We develop a novel network architecture that combines vector neurons and self attention enabling rotational invariance. The network is small in the number of hyper-parameters • We introduce a novel multi-headed self attention layer (VNAttention) that incorporates vector neurons through an adaptation that is simple to implement. • In this context we define novel feed forward linear layers that operate on vector representations (VNLinear)

Related Work
2D images are efficiently processed by deep neural networks due to their underlying regular grid. Unlike 2D images, point clouds are essentially unordered and scattered sets in 3D space, making it challenging to design neural networks for them. While order invariance has been widely explored for point cloud neural networks, rotational invariance has been just starting to gain popularity. Our previous work discussion is in the domain of 3D point cloud deep learning and focuses on rotation equivariant and invariant network architectures.
Rather than projecting or quantizing irregular 3D point clouds onto regular grids in 2D or 3D, researchers have designed deep network structures that ingest point clouds directly. PointNet [25] utilizes permutation-invariant operators such as pointwise MLPs and pooling layers to aggregate features across a set. PointNet++ [26] applies these ideas within a hierarchical spatial structure to increase sensitivity to local geometric layout. PointCNN [21] proposes to reorder the input unordered point clouds with special operators. Related to us, spherical convolutions were introduced [13] to address the problem of 3D rotation equivariance and attention has been applied to point graphs convolutions in GACNet [34].

Transformers and Self-attention
Transformers and self-attention were originally introduced for natural language processing Vaswani et al. [32], and have since become the state of the art there [38,9,7,42]. Direct application of selfattention to images would yield quadratic cost in the number of pixels. Thus, local multi-head self attention blocks were introduced to address image convolutions. Along this path, scalar dot-product self-attention within local image patches were introduced [16,28]. A family of feature vectors self-attention operators were showed in [48].
Sparse Transformers were suggested [5] to enable scalable approximations to global self-attention.
Similarly, blocks of varying sizes were utilized in [37] to scale attention.
To allow full self-attention on images, a set of patches are extracted from the image pixels to which all-to-all self attention is applied Cordonnier et al. [6], Dosovitskiy et al. [11]. Nevertheless, patch number is still high, making the model applicable only to small to medium resolution images. Similarly, Transformers were applied to image pixels after reducing image resolution and color space [4].
Since transformers and self-attention networks outperform convolutional networks on 2D images, our work harness their effective power to point clouds. In fact, the self-attention mechanism is a dot-product set operator and therefore seems particularly suitable for unordered point sets. There are a number of previous works that utilize attention for point cloud analysis [40,23,41,19]. Nevertheless, they apply attention on the whole point cloud, which introduces heavy computation and renders these approaches inapplicable to large point sets Recently, local vector self-attention was introduced [49] which enables scalability to large point sets.
Authors use vector attention however their vectors focus on feature representation while our vector neurons focus on rotational equivaiance.

Rotation Equivariant and Invariant Networks
Rotation invariance is a desirable property for tasks like shape classification and segmentation. The lack of robustness to rotation of classical deep learning architectures has driven interest for rotation invariant and equivariant designs.
Some works like LGR-Net [47] use surface normals in addition to the points coordinates. SFCNN [29] proposes an approach similar to multi-view by mapping input pointclouds to a sphere and performing operations on the sphere. Other works like [22,24] rely on more principled approaches borrowing tools from equivariant deep learning.
Recently multiple rotation equivariant deep learning architectures have been showed. For example, Qi et al. [25] achieved approximate pose equivariance by factoring out SO(3) transformations through object pose estimation. In general, equivariant networks are built on the theory of SO (3) representations [31,17,13,36,1]. Most of these works rely on the concept of convolution with steerable kernel bases. A steerable kernel basis is a family of functions undergoing a rotation in function/feature space given a rotation of their input [18]. Thus, features computed through these convolutions are SO(3) equivariant.
Recently, the universality of point cloud rotation equivariance has been studied in [12]. Vector Neurons were introduced in [8] enabling SO(3)-equivariant neural networks. Their work extends neural networks from 1D scalars to 3D vectors which allow mapping of SO(3) actions to latent spaces through definition of equivariant neural operations. Our work is inspired by Vector Neurons and takes a step further, incorporating them into attention layers to achieve rotational invariance.

Method
We introduce VNT-Net based on a combination between vector neurons [8] and multi-headed self attention (VNAttention). For this purpose, we define an extension to scaled dot-product attention [33] that provides SO(3) equivariance. In our work, the input is defined as where N is the number points, C is the number of channels in each point and D is the size of the VN in each channel (in our case D = 3). Given a weight matrix W ∈ R C ×C , we define a linear operation: f lin (·; W) operating on a vector-list feature #» V i ∈ V as follows: Neurons in standard artificial neural networks are essentially 1D scalars z ∈ R. When stacked in a list, these neurons form a However, when processing un-ordered data embedded in R 3 , like 3D point clouds, realizing the effect of SO(3) rotations applied to the input shape on these vector hidden layers is not obvious.
We are interested in constructing a rotation-equivariant self attention layer that commutes with the action of the SO(3) group. To achieve this, we adapt the neuron representation from a 1D scalar z ∈ R to a 3D vector v ∈ R 3 , leading to the definition of vector neuron. This results in a list of vector neurons This representation allows us to construct SO(3) equivariat self attention layers and finally an SO(3) invariant transformer network.
For this purpose we construct a self-attention vector neurons (VNAttention) layer and a vector neuron feed-forward network (VN-FFN) that satisfies rotation equivariance , namely, for any rotation matrix R ∈ SO (3): Figure 3: VNAttention attention matrix rotation invariance demonstration.
Self-attention is the central layer of any transformer network. We base our VNAttention layer on scaled dot-product attention [33](see Figure 2 right): In the case of vector neurons, we define the parameters in the following way: We define #» X (c) as the indexing of a channel c in , ..., C}). Finally, we define VNAttention as follows: For simplicity, we easily show that in the case where D = 1 VNAttention boils down to the original Attention: Figure 3 demonstrates the equivariance and invariance to the rotation group of the attention matrix (red square). Given an input (left airplane) and a rotated one (right airplane), the attention layer structure addresses rotation as commutative (bottom part) and thus is equivariant to rotation group.

Flattening
In order to optimize the VNAttention layer, we flatten its vector neurons input. This allows performing a single matrix multiplication instead of numerous dot products as summation of row-col multiplications. Thus we flatten the #» Q and #» K matrices as follows: In this way we don't need to save intermediate representations of S (c) .

Multi-Headed VNAttention layers
As is the common practice in Transformer networks, we found that it is beneficial to use multi-headed attention, instead of performing a single attention function with d model -dimensional keys, values and queries.
We achieve this by linearly projecting the queries, keys and values h times with different learned linear projections to d k and d v dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding d v -dimensional output values. These are concatenated and once again projected.
The following equations describe this process mathematically:

VN Feed-Forward layer
We found out that using a feed forward layer is beneficial after the multi-headed layers. Thus, after each multi-headed VNAttention block we add a VN Feed-Forward layer as defined in [8].

Vector Transformer Network
The basic building block of our network is constructed as follows: Multi-Headed VNAttention block, Residual connection, VN Feed-Forward block, and Residual connection. We construct our network from 3 stacks of basic blocks, that is, the input for each block is the output of the previous block. After which, we concatenate the output of the 3 blocks, then we pass it to an Invariant Layer, similar to the layer used on VNN [8], to get an SO(3) invariant representation of the output. The last part of the network is a simple classification head to get scores (see Figure 2 (left figure)).
For part-segmentation setups, we use the same architecture with 2 small changes. We pass the categorical data through a simple linear layer, then concatenate it's output to the output of the Invariant Layer. We change replace the classification head with a simple segmentation head.

Linear Dependency Problem
Similar to the observation in [8], when encoding data across network layers, thus resizing it, vectors may become linearly dependent due to their duplication. As a consequence, the output of a linear layer applied to them would degenerate to a set of linearly dependent vectors n R C×3 (pointing to the same direction). This is analogous to applying a per-pixel 1x1 convolution to a gray-scale image (single input channel).
To deal with this issue, we add to our architecture an edge convolution block in the input layer, mapping features from R 1×3 to features in R C×3 with C > 1, and those are put back to the basic block stack.

Results
We evaluate our model on two core tasks in point cloud processing: classification and segmentation. This shows that our network works well on diverse types of tasks. In this section we discuss our datasets, network implementation and performance, result and comparisons.

Datasets
We employed the ModelNet40 [39] and ShapeNet [39] datasets for evaluation. The ModelNet40 dataset consists of 40 classes with 12,311 CAD models in total. In the classification setup, we used 9,843 CAD models for training and the rest for testing.
For segmentation on the ShapeNet dataset, we followed [43] by using ShapeNet-part for part segmentation, which has 16 shape categories with more than 30,000 models.

Rotated Dataset
In both classification and segmentation, we train and test our network on both the original dataset as well as its rotated versions denoted as follows: z/z, z/SO(3), and SO(3)/SO(3). z stands for data augmentation with rotations only around the z-axis and SO(3) stands for arbitrary rotations. All rotations are generated randomly at training time.
The notation A/B denotes the training data was A and the testing data was B. The aim here was to evaluate the capability of our network as well as SOTA networks to train and test on similar sets z/z and SO(3)/SO(3) and to generalize on z/SO(3).

Data Augmentation.
In the implementation of our network, we found that not all common data augmentation technics are useful. In this section, we will describe what augmentations were useful and what didn't work.
Sampling. During the training phase on Modelnet40, we used run farther point sampling on the training data. Where we sample 1024 points on every model randomly each time. we found this help us to avoid overfitting the training set. The reason we found it more useful than just random sampling was that this sampling creates more variety of vectors directions. This will make the attention matrix to be more sparse and find more diverse connections.
Randomly dropping points by changing their value of them to zero. This augmentation can be found used in most of the implementation of the comparable network. This augmentation didn't help and in 't help and most of the time caused our implementation to not converge.
Normalization. Using data that has its scale normalizes gives us much more stable training. In a case, where scale normalization was not utilized the network frequently didn't converge or in the middle of the training stop converging or even got much worst on the test set. The normalization is done by subsubsection of the centroid of the point cloud and dividing by the maximum vector length from all the points inside the point cloud.
Scale. Random scaling augmentation is useful in the range of 0.8 to 1.25. This is even when the VNAttention attention matrix is not affected by the size scaling because of the softmax operation.
Shift. Random Point shifting was useful to add noise to our network and avoid overfitting. We randomly shift all our points in the range of -0.1 to 0.1.

Network implementation
The principal hyper-parameters of our network are as follows: 1. Linear Dimension: The feature dimension of the output of the edge convolution, as well as that of the input/output of each MultiHeaded VNAttention and Feed Forward layer.

Heads Number:
The numbers of heads in each MultiHeaded VNAttention layer.
3. Head Size: The feature dimension of each head in each MultiHeaded VNAttention layer.
For classification, the setup that worked best for us is: Linear Dimension=16, Heads Number=24, Head Size=16. For segmentation, the setup that worked best for us is: Linear Dimension=128, Heads Number=14 and Head Size=16.

Classification Evaluation and Comparison
We evaluate classification results on ModelNet40 comparing with VN networks [8], and SOTA point transformer networks [49] [15] and SOTA rotation invariant networks (see Table 1).
The table is divided into sections top to bottom as follows. First are vector neurons networks and DGCNN, next are point cloud convolution networks, following are rotational-sensitive and invariant networks, transformers and finally networks using points+normals representations as input.
The highest accuracy network for z/z is Shellnet [45] however not rotational invariant.

Semantic Segmentation Evaluation and Comparison
We evaluate semantic segmentation results on the ShapeNet part-segmentation in average category mean IoU 2 and global mean IoU 3 where Training is done on aligned data without rotation augmentation. The

Conclusions
In this work we combine recently introduced vector neurons with self attention layers to build a point cloud vector neuron transformer network (VNT-Net) that is rotational invariant. For this purpose, we develop a novel multi-headed self attention layer (VNAttention) that incorporates vector neurons through an adaptation that is simple to implement. The network is small in the number of hyper-parameters and results show high accuracy that is competitive with SOTA networks that are significantly larger in terms of number of hyperparams. Although our network does not achieve the highest accuracy it provides an interesting set of simple tools to move transformers into rotational invariant and equivariant domains. In future work, we plan to take this approach to higher dimensions and adapt vector neurons to new transformer networks.

A VNAttention Proof and Analysis
Proof of VNAttention rotational Equivariance We will show that R ∈ SO(N ) commutes with the VNAttention layer (Equivariance):

Analysis of the attention weights
The result of the dot product of #» Q (c) · #» K T (c) can also be represented in the VNAttention( s ij = q i · k T j = |q i ||k j | cos(θ) For the Matrix S we get the following equation: S ij = c∈C q (c),i · k T (c),j = c∈C |q (c),i ||k (c),j | cos(θ) This equation show that the closer the VNs directions, the larger the weight value will be. This is very similar to classic self attention, where the more correlated the features are the larger the weight will be.

B Classification and Segmentation Heads
Classification Head. The classification head is built form 3 linear layers, and we add drop out and leaky-ReLU between the layers. Dropout is set to 50%.
Segmentation Head. The segmentation head is built from 4 linear layers, and between each layer we add ReLU and batch-norm.

C Extra Hyper Parameters
Here will describe extra hyper Parameters that where used in the training of the model.
Learning Rate and Optimizer. We found that using Adam Optimizer with learning rates in the range of 1e-3 to 1e-4 achieves good result. The best results on segmentation and classification were achieved using a learning rate of 5e-4.
Batch Size. We used a batch size of 8 on GPU of GTX 1080Ti and 16 when using RTX 3090.
Scheduler. We use learning rate step scheduler with step size of 20 and γ = 0.9.