{'fraction_non_alphanumeric': 0.02630681818181818, 'fraction_numerical': 0.012102272727272727, 'mean_word_length': 5.021553198768388, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-supervised Learning of Point Clouds via Orientation Estimation

Point clouds provide a compact and efficient representation of 3D shapes. While deep neural networks have achieved impressive results on point cloud learning tasks, they require massive amounts of manually labeled data, which can be costly and time-consuming to collect. In this paper, we leverage 3D self-supervision for learning downstream tasks on point clouds with fewer labels. A point cloud can be rotated in infinitely many ways, which provides a rich label-free source for self-supervision. We consider the auxiliary task of predicting rotations that in turn leads to useful features for other tasks such as shape classification and 3D keypoint prediction. Using experiments on ShapeNet and ModelNet, we demonstrate that our approach outperforms the state-of-the-art. Moreover, features learned by our model are complementary to other self-supervised methods and combining them leads to further performance improvement.


Introduction
As a concise and expressive form of representing 3D shapes, point clouds are ubiquitous in various 3D vision tasks. Massive point cloud datasets have become available with the development of 3D scanning devices and techniques. ShapeNet [9], ModelNet [86] and ScanNet [11] are instances of such datasets. Models such as Point-Net [58] have achieved impressive results on these datasets. However, manually annotating these large amounts of data is costly and impractical to scale. This raises the need for methods that can learn from unlabelled data to reduce the number of annotated samples required for learning. Recently, self-supervised learning have shown promising performance on several computer vision tasks. Selfsupervision is a learning framework in which a supervised signal for a pretext task is created automatically, in an effort to learn representations that are useful for solving realworld downstream tasks. * Indicates equal contribution A simple form of self-supervision is predicting rotations [20]. This approach has achieved state-of-the-art results on image classification [35]. In this paper we propose the first self-supervised approach based on rotations of 3D data. Among various 3D representations, we focus on point clouds as an efficient representation in which rotations can be easily applied via matrix multiplication. Manually annotating point clouds to provide information such as category labels or keypoints is cumbersome and costly. Notably, most public 3D models have a canonical orientation, or at least a known up axis. We can leverage this orientation information to provide a label-free supervision signal for downstream tasks. More specifically, we can arbitrarily rotate an input point cloud and train a model to recover the rotation angle. The auxiliary task of predicting these rotations requires high-level understanding of 3D shapes, since canonical orientation of the object implicitly encodes important information about its functionality [16].
We demonstrate that a model trained for this proxy task generates semantic features useful for target tasks such as classification and keypoint prediction. We show that simple models such as SVMs can be trained on these features and achieve strong performance even when trained on a small subset of the training data. Hence, our approach facilitates learning with fewer labels, and obviates the need for manually annotating massive datasets. Finally, we discuss how our approach can be combined with other self-supervised models to achieve further performance gain, indicating that our learned features provide complementary information to other methods.

Deep Learning on Geometry
There has been a recent surge of interest on learningbased approaches on geometric data [6,10,64,84,86]. These models are able to encode rich prior information about the space of 3D shapes which helps to resolve ambiguities in the input. Typical convolutional architectures require highly regular input data formats such as image grids or 3D voxels. Unlike images, geometry usually does not have an underlying grid structure, requiring new building blocks replacing convolution and pooling or adaptation to a grid. As a simple way to overcome this issue, view-based [72,83,57] and volumetric representations [34,43,74,86] or their combination [59] place geometric data onto a grid. Voxel representations are a straightforward generalization of pixels to the 3D case. however, the memory footprint of these representations grows cubically with resolution. Data adaptive representations such as octrees [66,74] mitigate this issue, yet they lead to complex implementations and are still limited to relatively small voxel grids.
Point clouds provide a more efficient representation by only focusing on surface points, yielding a tensor with a fixed number of 3D coordinates [1,15]. Point clouds as an input modality present a unique set of challenges when building a network architecture. The grid structure is not available in raw point clouds as unordered sets of points. To address this issue, Qi et al. propose PointNet [58] which respects permutation invariance of point clouds by using a max-pooling operation to form a single feature vector representing the global context. PointNet++ [60] extends Point-Net by recursively applying it on a nested partitioning of the input point set. Kernel correlation and graph pooling are proposed for improving PointNet-like methods in [70]. PointCNN [39] leverages spatially-local correlations using a χ-Conv operator that weights and permutes input points and features before they are processed by a typical convolution. PointConv [85] and Relation-Shape CNN [41] also focus on local structures of point clouds and further improve the quality of captured features. Recently, graph convolutional networks have shown promising results on learning tasks on point clouds. Dynamic Graph CNN [81] applies a graph convolution to edges of the k-nearest neighbor graph of the point clouds, which is dynamically recomputed in the feature space after each layer. While the aforementioned models achieve great performance on various tasks, they require large number of annotated samples for training. Hence, there is a need for efficient methods with less supervision.
Several papers consider predicting the spatial relation between two randomly chosen image patches [13,46,47,49]. An extension of this idea to 3D is presented in [67]. Our work bears a resemblance to [20] which randomly rotates an image by one of four possible angles and let the model predict the rotation. [50] proposes to predict future patches in representation space via autoregressive predictive coding. An overview and comparison of various 2D self-supervised learning algorithms is provided in [35,4].
Recently, various methods have been proposed for unsupervised learning of point clouds. Autoencoder-based approaches learn features through reconstructing the point cloud data [1,17,89,94]. Generative models learn representations in the process of generating plausible point clouds [37,73,75,84]. Concurrent to our work, various pretext tasks have been proposed for self-supervision on point clouds. Sauder et al. propose to learn features by reconstructing point clouds whose parts have been randomly rearranged [67]. Zhang et al. train deep graph neural networks to solve two proxy tasks, part contrasting and object clustering [91]. A multi-task learning framework is proposed in [24] which learns features by optimizing three different tasks including clustering, prediction, and reconstruction. Our approach requires more holistic understanding of shapes, and is relatively easy to implement. As we demonstrate in experiments, our model outperforms several competing methods, and can also be combined with them to achieve further performance gain.

Method
Inspired by the success of self-supervised learning on images, we propose a self-supervised method to learn representations of point clouds. We consider the proxy task of rotation prediction. A set of K rotations R = {R 1 , R 2 , . . . , R K } is used, where R i (X) rotates the input point cloud X to the i th angle. We train a classifier F (·)  to predict the rotation applied to the input point cloud. The model takes a transformed point cloud R i (X), with R i randomly selected from R, and outputs a probability distribution over all possible rotations. We train the model using cross-entropy loss in a supervised manner.
Alternatively, we can train a model to regress the angles instead of classifying them. More specifically, we first randomly select an angle by sampling coordinates x, y, z ∼ N (0, 1) and normalizing the resulting vector (x, y, z) T to have a unit norm. It can be shown the resulting vectors are uniformly distributed on the unit sphere. The input point cloud is rotated by this angle, and a regression model is trained to regress the axis of rotation and the rotation angle with respect to this axis. We train the model with L 2 loss using a linear combination of axis and angle prediction losses with equal weights. Figure 2 illustrates our architecture. Our models are based on PointNet [58] and DGCNN [81] in which the final layers are modified for rotation classification or regression. The classifier outputs K probabilities and the regres-sion model outputs four real numbers (three for the rotation axis and one for the rotation angle). The features from the rotation prediction model are then used for several target tasks including classification and keypoint prediction. ShapeNet [9], a large scale dataset of 3D shapes, is used to train the rotation estimation model. 3D models in ShapeNet are annotated with upright and front orientations, allowing us to align them to a canonical orientation.
Number of angles K is a hyper-parameter. Using a small K gives a high accuracy on rotation prediction while might not yield the best features for downstream tasks since predicting a small number of rotations is a simple task. On the other hand, a very large K gives a low accuracy on rotation prediction as angles become closer to one other. Table 1 shows the rotation classification accuracy for different values of K. As we observe, the accuracy decreases for larger K.
For a given choice of K we need to select the angles as uniformly as possible to avoid bias in learning. Ideally, we can consider a regular polyhedron with K vertices.   Figure 1 illustrates the distribution of angles for various K values.
Finally, we use the trained rotation estimation model to extract features for downstream tasks. These features are passed to a linear SVM for shape classification. In the keypoint prediction task, we modify final layers of the angle prediction model and fine-tune it for keypoint regression. We train the rotation estimation and target models on ShapeNet and ModelNet respectively.

Experiments
We demonstrate that features learned by the rotation prediction model are useful for downstream tasks. We consider two tasks: shape classification and 3D keypoint prediction.

Shape Classification
We consider the task of shape classification on Model-Net [86]. Using the rotation prediction model trained on ShapeNet, we generate features for ModelNet-40's train split. We use the global activations after the max pooling layers of PointNet and DGCNN as we found them to yield the best performance. A linear SVM classifier is trained on these activations for the task of predicting the shape category. This experiment evaluates the learned features in a transfer learning task, demonstrating their generalizability. Figure 2 illustrates the architecture. Note that the rotation estimation model is trained on the ShapeNet dataset while the linear SVM is trained on features from ModelNet. Table 2 shows accuracy of the linear SVM on the test set of ModelNet-40. It also compares our model's performance with various competing methods. As we observe, our approach outperforms competing methods. Note that the accuracy depends on the number of angles, with K = 18, 32 achieving the best performance. Using too large or too small values for K leads to inferior performance as the rotation prediction task becomes too difficult or too easy.
We can also combine our approach with other selfsupervised learning methods. More specifically, we concatenate the features learned by our model with features extracted from the context prediction model [67] and train the SVM classifier on the combined features. As shown in Ta-  Previous work VConv-DAE [69] 75.50% 3D-GAN [84] 83.30% Latent-GAN [1] 85.70% FoldingNet [89] 88.40% VIP-GAN [23] 90.19% Context Prediction (DGCNN) [ Table 3: Test accuracy on ModelNet-40 shape classification using the pretext task of rotation regression on ShapeNet.
ble 2, this leads to improved performance compared to individual models, demonstrating that our approach learns complementary features to other self-supervised methods. Intuitively, our rotation estimation model learns global orientation information while [67] learns part-level information required for the task of context prediction. Both of these information sources are useful for the shape classification task, therefore combining them leads to improvements over each individual model.
We can also estimate rotations via regression instead of classification. As described in Section 3, in this approach we aim to regress the exact rotation angle rather than a discretized approximation of it via K pre-selected angles. We train a rotation regression model on ShapeNet and use it to extract features from ModelNet samples. A linear SVM classifier is trained on these features. Table 3 shows the accuracy on ModelNet-40's test set. We observe that features learned from the regression model achieve inferior performance compared with the classification model. Regressing the exact rotation angle is a challenging task. It resembles our classification setting with K → ∞ which does not yield the best performance. The learned features become too task-specific and not as transferable to a shape categorization task.
We also study dependence of our model on the size of the training set. We vary ModelNet-40's train set size for the linear SVM. A certain percentage of data is randomly sampled and used to train the classifier. We then evaluate the model's accuracy on the full test set of ModelNet-40. Figure 3 depicts the linear SVM's classification accuracy as size of the training set varies from 1% to 100%. The results indicate that we still have a relatively strong perfor- mance even when a small number of annotated samples are available. The features learned from the rotation prediction model contain information useful for the classification task. Therefore, a simple classifier trained with moderate supervision suffices to achieve high accuracy.

3D Keypoint Prediction
We also consider the task of 3D keypoint prediction. We use the training data provided by [90] which includes 10 keypoints for each object in ShapeNet's chair category. We pre-train a model for rotation prediction and fine-tune it for keypoint regression. More specifically, we first train a rotation prediction model (on all categories) and then modify its final layers to regress 10 keypoints for each input shape. The regression network is initialized with weights of the rotation prediction model and then fine-tuned with chamfer loss between predicted and ground truth keypoints. At inference time, we map each regressed point to its nearest neighbor point in the input point cloud. PCK 1 curves for our approach are shown in Figure 4, and are compared with [90] and [25]. We observe that our method outperforms competing methods for all error thresholds. Note that compared with the shape classification task, keypoint prediction relies more on local and partlevel information. Our results indicate our self-supervised approach can transfer useful local and global information. We also visualize the keypoints in Figure 5. We see that predicted and ground truth keypoints are close to each other and in many cases they overlap. 1 Percentage of Correct Correspondences

Conclusion and Future Work
We present a self-supervised approach for learning representations of 3D point clouds. A supervised signal for the pretext task of rotation prediction is created to learn representations that are useful for solving target tasks. We devise models for learning proxy and downstream tasks, and demonstrate results on shape classification and 3D keypoint prediction. The results indicate that our approach achieves superior performance to competing methods. Each selfsupervised learning method learns features tailored for its own proxy task. We demonstrate that features from different models can complement one another and outperform each individual model. Finally, we show that only a small subset of the training data suffices to achieve high accuracy on shape classification, eliminating the need for annotating massive 3D datasets.
While our work focuses on point clouds, self-supervised learning can also be applied to other 3D representations such as meshes, voxel grids and implicit functions. Each representation has its own merits and disadvantages. Studying self-supervision for different representations is relatively under-explored and devising efficient proxy tasks for various 3D representations is a promising venue for future work. We can also consider combining self-supervised techniques for different representations. Finally, evaluating our model on other downstream tasks is another direction for future research.