{'fraction_non_alphanumeric': 0.0373048307660065, 'fraction_numerical': 0.01907157078486849, 'mean_word_length': 5.029907329401853, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Standard Transformer Models for 3D Point Cloud Understanding with Image Pretraining

While Standard Transformer (ST) models have achieved impressive success in natural language processing and computer vision, their performance on 3D point clouds is relatively poor. This is mainly due to the limitation of Transformers: a demanding need for large training data. Unfortunately, in the realm of 3D point clouds, the availability of large datasets is a challenge, which exacerbates the issue of training ST models for 3D tasks. In this work, we propose two contributions to improve ST models on point clouds. First, we contribute a new ST-based point cloud network, by using Progressive Point Patch Embedding as the tokenizer and Feature Propagation with global representation appending as the decoder. Our network is shown to be less hungry for data, and enables ST to achieve performance comparable to the state-of-the-art. Second, we formulate a simple yet effective pipeline dubbed \textit{Pix4Point}, which allows harnessing Transformers pretrained in the image domain to enhance downstream point cloud understanding. This is achieved through a modality-agnostic ST backbone with the help of our proposed tokenizer and decoder specialized in the 3D domain. Pretrained on a large number of widely available images, we observe significant gains of our ST model in the tasks of 3D point cloud classification, part segmentation, and semantic segmentation on ScanObjectNN, ShapeNetPart, and S3DIS benchmarks, respectively. Our code and models are available at: \url{https://github.com/guochengqian/Pix4Point}.


Introduction
Point clouds are one of the most essential 3D representations, with broad applications in robotics, autonomous driving, medical imaging analysis, etc. However, point clouds are quite expensive to acquire, with data cleaning and annotation costs being even higher [23]. Thus, despite the impressive performance gains made possible by deep learning [21,24,38,39], their need for massive labeled training  1 have achieved great success in natural language processing and computer vision [11,12], they result in relatively poor performance in point cloud understanding [37,63]. As a concrete example, ST only reaches 77.2% overall accuracy in real-world point cloud classification [50], meanwhile, state-of-the-art convolutional neural networks (CNNs) [35,42] on point clouds attain much better accuracy (above 85%).
To improve the performance of Transformers on point clouds, previous methods mainly rely on integrating convolutions into Transformers such as PCT [17] or conducting self-attention locally like Point Transformer [65]. These approaches make the Transformers specialized for point clouds rather than being ST-based. However, ST models are highly preferred. First, ST can process data from all modalities, including language, images, and point clouds [7,12,51,63]. Recent work, such as [45], shows that ST can serve as a generalist agent for more than 600 tasks across domains. This cross-modality ability is unique to ST rather than CNNs or hybrid models. Second, ST is shown to benefit more from big data in natural language processing and computer vision. With increasing training data, ST tends to overfit less and is able to perform better than its CNN counterparts [11,20]. This is the reason why almost all big models today are STbased [2,11,13,44]. Previous studies on point cloud Transformers diverge from this ST trend (generalist agents and big models). Empowering Standard Transformers to perform well in point cloud understanding is of practical importance but remains under-studied. In this work, we are instead especially interested in ST models for point clouds. We ask: How can the performance of Standard Transformers be improved for point cloud understanding ? We attempt to answer this question from the following perspectives: (i) The tokenizer and decoder used with ST could be improved. Although the Transformer backbone should be kept as standard, the tokenizer and the decoder of the network can be specialized to the task. We find that progressively projecting input point clouds into tokens results in significant performance gains compared to the single projection layer counterpart. This progressive design only modifies the tokenizer, while keeping the ST backbone untouched. Also, we observe that appending global representations in the decoder, such as the maxpooled tokens, boosts performance for both 3D classification and segmentation.
(ii) 3D point clouds are expensive to acquire, but other rich domains (e.g. images) could be exploited for 3D point cloud tasks. In contrast to point clouds, 2D images are much cheaper and easier to collect/annotate. For comparison, Mod-elNet [56], the widely used point cloud classification dataset, consists of only 12, 311 CAD models. Meanwhile, Ima-geNet [10] is over 100 times larger with more than one million (1,331,167) images. In addition, there are an uncountable number of public images on the Internet, while the available point cloud data is rare. Hence, there is a need to study the possibility of exploiting learning from images to better understand 3D point clouds.
Our solution to the second perspective is Pix4Point, which is a simple yet effective pipeline that allows utilizing an image-pretrained Transformer to understand point clouds in their native format. As illustrated in Fig. 1, Pix4Point pretrains an ST backbone on a large amount of tokenized (i.e. patchified) images (e.g. from ImageNet [10]) and then finetunes this image-pretrained Transformer in point-cloud tasks with tokenized point clouds as input. To the best of our knowledge, Pix4Point is the first pipeline to directly leverage the weights learned in the image domain to improve the performance for 3D point cloud tasks without any change in the network's backbone or using image projections. Contributions: (i) We present a new ST-based network called Point Vision Transformer (PViT). PViT improves ST on point clouds [63] with a progressive point patch embedding (P3E) tokenizer that gradually projects a given point cloud into tokens by graph convolutions, and a decoder using Feature Propagation with global representation appending. (ii) We propose Pix4Point, a simple yet effective framework that facilitates image pretraining for point cloud understanding. Our method enables significant performance improvements of ST in 3D tasks on various benchmarks: ScanObjectNN [50], ShapeNetPart [6], and S3DIS [4].  [21,24,38,39], current prevailing point cloud processing methods are entirely data-driven and consist of three main approaches: view-based [15,18,47], voxel-based [9,16,36], and point-based [38,39]. Among these three approaches, point-based methods, which directly take point clouds as input and process in their unstructured format, are receiving increasing attention, since there is no information loss. PointNet [38], the pioneering work of point-based methods, proposes to model the permutation invariance of points by restricting feature extraction to be pointwise. PointNet++ [39] improves PointNet by further capturing local geometric structures. Most current point-based methods focus on the design of local modules [26, 27, 29, 40-42, 48, 53, 55, 65]. In this work, we pay attention to point-based methods, specifically, Transformerbased networks [51], which have been proposed as a new paradigm for processing point clouds [17,63,65]. 2D to 3D Transfer Learning. A straightforward means of transferring 2D learned representation to 3D is to use view-based methods. As a specific example, PointCLIP [64] showed that a pretrained CLIP [43] model can be used directly for zero-shot point cloud classification. More advanced methods leverage point-pixel correspondences [19,30,54] between point clouds and multi-view images, where pointwise pseudo-labels can be generated from images to optimize a point cloud network. More recently, Image2Point [59] presents a kernel inflation technique that expands 2D kernels of a 2D CNN into 3D kernels and further applies them to voxel-based point cloud understanding. However, these approaches lead to performance degradation. Multi-view methods can lead to non-ideal views, and a loss of geometric information [3,18], while kernel inflation only shows insignificant performance gains given the gap between 2D and 3D kernels [59]. In this work, we present a novel pipeline that can directly utilize the Transformer pretrained on images for 3D point cloud tasks, without image projection or kernel inflation. Self-Supervised Learning on Point Clouds. One prominent way to tackle the need for large labeled data is to rely on self-supervised learning (SSL) [5,8]. Previous works in SSL for point clouds rely on pretext tasks [1,46] or auto-encoders [1,28,52,60]. PointContrast [58] proposes to generate two views of the point cloud with random transformations and ask the neural network to minimize the distance between matched points. On the other hand, Point-BERT [63] and Point-MAE [37] propose masked auto-encoders to learn useful representations for 3D classification. Our method does not count on the availability of large amounts of point cloud data, as usually needed in SSL. Instead, we focus on how to utilize already available image data for pertaining point cloud networks.

Transformers
Specialized Transformer Architectures. Based on its multi-head attention block, the Transformer [51] is believed to be the most successful architecture for natural language process (NLP). Vision Transformer (ViT) explores the direct application of this Transformer on image patches to solve vision tasks [12]. There is also an increasing interest in Transformer-like architectures for point cloud processing. Unfortunately, ST [12,51] cannot achieve results comparable to their convolutional counterparts in point cloud tasks [63], mostly due to the small scale of point cloud data. This leads to a boom of specially designed variants of Transformerbased models [17,22,25,61,65]. In contrast to these previous works that specialized the Transformer architecture, we maintain the same ST architecture and study means of improving its performance, by only changing its tokenizer and decoder. Transformer as a General Backbone. [34] proposes a pretrained Transformer as a universal backbone for protein prediction and language understanding. Gato [45] is proposed to be a generalist agent that can perform tasks ranging from image captioning and language tasks to robot manipulation and reinforcement learning. Most recently, Omnivore [14] proposes a vision Transformer model that works with multiple modalities such as videos and RGBD data, showing the benefit of 2D training in improving performance in different modalities. In our work, we utilize 2D image pretrained Transformers. Unlike Omnivore and Gato, our goal is to leverage these image-pretrained Transformers for point cloud tasks, which is under-studied.

Methodology
This section first introduces a new ST-based network called PViT that improves the performance of ST for point cloud understanding. Then, we present Pix4Point, a practical framework that utilizes plentiful available image data to help point cloud tasks. The architecture of PViT and the full pipeline of Pix4Point are illustrated in Fig.2.

PViT Network
Inspired by ViT [12], Yu et al. designed the first STbased point cloud network in their PointBERT work [63], which we refer to as vanilla ST throughout texts. Vanilla ST presents a simple point tokenizer that groups the input point cloud into local patches by a single farthest point sampling (FPS) operation to sample center points, a k-nearest neighbor (kNN) layer to group neighbors, and a mini-PointNet [38] to project local patches to point tokens. This simple extension of ViT [12] to point clouds achieves rather poor performance in several 3D benchmarks. A follow-up work named Point-MAE uses the same vanilla ST model as Point-BERT, but relies on a better SSL method to improve the performance. In this work, we propose a new ST model for point clouds called Point Vision Transformer (PViT) that can achieve comparable performance to the state-of-the-art. PViT consists of three modules: a tokenizer, an ST backbone, and a task decoder. While the backbone architecture is kept as an unchanged Standard Transformer, the tokenizer and the decoder are specialized to the domain and task. We describe our improved tokenizer and decoder in PViT next. Tokenizer t: Progressive Point Patch Embedding. Consider input point cloud X ∈ R N ×(3+Cin) representing N points with C in input features (e.g. colors, normals), the tokenizer t projects the input point cloud into a sub-cloud (a smaller number of points but in a higher dimensional space). Our Progressive Point Patch Embedding (P3E) progressively tokenizes point clouds through multiple stages with t: At each stage, P3E uses FPS to sample a subset of points with a fixed sample ratio (1/4), queries neighbors for each downsampled point, and leverages graph convolutions to abstract features to sub-clouds. P3E consists of two FPS stages rather than one in Point-BERT [63] and Point-MAE [37], which is shown to ease the optimization and offers large performance gains in various experiments. The graph convolution in each stage of P3E is defined as Eq. 1: where p i and x i denote the coordinates and features of the ith center point. The index j defines the j-th neighbor of point i. The proposed graph convolution extracts features for each center point from the relative positions and relative features of its neighbors. The maxpooled features inside each neighborhood, MAX j:(i,j)∈N x ij , enrich the tokens with global information from the local groups. Both h 1Θ and h 2Θ are two-layer MLPs with batch normalization and ReLU. Decoder g: Feature Propagation with Global Representation Appending. The goal of decoder g is to yield the desired output for each task. For segmentation, we introduce Feature Propagation, a successful and widely used decoder design from PointNet++ [39], into the ST model. Feature Propagation gradually interpolates the feature map and concatenates it with point features from the tokenizer to produce

Pix4Point Pipeline
Another main argument of this work is that 3D point clouds are expensive to acquire, but images are easily available and could be exploited. To this end, we propose Pix4Point, a framework that enables improving PViT through image pretraining. We illustrate the pipeline of Pix4Point in Fig. 2. It consists of three stages: (1) pretraining on images, (2) transferring image-pretrained weights to PViT, and (3) finetuning PViT on point cloud tasks. Pretraining Stage. The goal of this stage is to utilize a large number of public images to pretrain the ST backbone of PViT. By default, we pretrain ViT-S using the the selfsupervised method MAE [20] on ImageNet-1K [10] dataset. Note that our Pix4Point is generally agnostic of the pretraining strategy. Besides the aforementioned self-supervised pretraining, there are many other applicable strategies, e.g. supervised training through DeiT [49]. Refer to Sec. 5 for the effects of different pretraining strategies. Weight Transfer Stage. The goal of this stage is to transfer the image-pretrained weights to the downstream point cloud network. Unlike previous methods that require 2D view projections or kernel inflation [47,59], this stage in Pix4Point is as simple as using the weights of the imagepretrained Transformer as an initialization for the backbone of our PViT network. This is possible because the exact same ST backbone (e.g. ViT-S backbone) is used in both the image pretraining and the downstream point cloud tasks. Finetuning Stage. The finetuning stage finetunes the imagepretrained ST backbone F for point cloud tasks. The tokenizer t and the task-specific decoder g of PViT are jointly trained from scratch during this fine-tuning stage. The loss for Pix4Point can be described as follows: where {y i } B i=1 defines the labels of a mini-batch of input point clouds X and L is a Cross-Entropy (CE) loss. Why does Pix4Point Work? The 2D tokenizer and the 3D tokenizer project the input images and the point clouds into a shared token space R C , respectively, where each token from 2D or 3D is embedded as a vector of the same length C. Despite the domain shift between 2D and 3D and owing to the diverse large-scale image data in the pretraining stage, the ST backbone has learned to transform and mix the information of tokens in this shared token space. Therefore, it only takes a few finetuning epochs to tune ST and optimize the domain-specific tokenizer for the downstream point cloud tasks. Interestingly, we show (see Sec. 5) that even when the image pretrained Transformer backbone is kept frozen, our proposed pipeline can still adequately recognize point clouds by training only the tokenizer and decoder in the downstream task. This verifies our hypothesis that how to transform and mix information in the shared token space has been learned during image pretraining stage. Comparison with Recent Methods. The state-of-the-art methods for improved Standard Transformers on point clouds are Point-BERT [63] and Point-MAE [37]. Both methods find that the ST cannot achieve comparable results in point cloud tasks. They utilize self-supervised learning methods on point cloud data to improve performance. In our work, we diverge from previous studies, and propose two contributions: (i) the PViT network that improves ST on point clouds through the P3E tokenizer and Feature Propagation decoder with global representation appending; (ii) the Pix4Point framework that exploits image pretraining for 3D point cloud understanding.

Experiments and Results
We conduct extensive experiments on various 3D benchmarks, including semantic segmentation on S3DIS [4], part segmentation on ShapeNetPart [6], and real-world point cloud classification on ScanObjectNN [50] to verify the strengths of the PViT network and the Pix4Point framework. In all benchmarks, we highlight the comparison among ST-based methods: PViT, PViT+Pix4Point (PViT with an image-pretained backbone), the vanilla ST model by Point-BERT [63] (i.e. Point-BERT trained from scratch), Point-BERT [63], and Point-MAE [37]. All of these methods use the same ST backbone (ViT-S) with 12 self-attention blocks, 384 channel size, and 6 heads for all tasks.

Training Setup
Pretraining setup. We pretrain ViT-S [12] using MAE [20] on ImageNet-1K [10] in a self-supervised fashion. The weights of the ViT-S backbone from the final model (epoch 400) are used to initialize the ST of our PViT network. Finetuning Setup. For segmentation on S3DIS, PViT is trained using the Cross-Entropy loss with 0.2 label smoothing, AdamW optimizer [33] with a learning rate of 1e-4, a cosine learning rate scheduler, warmup epochs of 10, a weight decay of 1e-4, a batch size of 8, and for 600 epochs. Data augmentation includes rotation, scaling, jittering, color auto-contrast, and color dropping. For part segmentation on ShapeNetPart, PViT is trained similarly as on S3DIS using 2, 048 points as input. The difference is: the models are  nificant margin (+4.4 mIoU). This shows the effects of our proposed P3E tokenizer and Feature Propagation decoder with global representation appending. (3) Most importantly, results demonstrate that image pretraining using Pix4Point impressively further improves PViT by 5.2% in mean IoU (mIoU) and 5.3% in mean accuracy (mAcc). This achievement is simply due to initializing the ST backbone with an image-pretrained Transformer. With image pretraining, we show that an ST-based network can perform much better than representative 3D methods including PointNet++ [39], PVCNN [32], and KPConv [48]. Compared to Point Transformer [65], a highly optimized local transformer-based network with a hierarchical structure, PViT with image pretraining can reach achieve comparable performance to it, while using only the Standard Transformer as the backbone. These observations verify our argument: progressive tokenizer and global appending improve the performance of ST, and image pretraining can be beneficial for point cloud understanding. Qualitative Results. Fig. 3 shows some qualitative results of PViT and PViT+Pix4Point. The latter yields more precise segmentation maps than the former. Closeups show that image pretraining helps PViT successfully segment the boards (1 st and 3 rd rows), the door (2 nd row), and the columns (2 nd and 3 rd rows). PViT also achieves better qualitative results than Point-BERT [63].

3D Part Segmentation
Dataset. ShapeNetPart [62] is a richly annotated 3D dataset of 16 shape categories selected from the ShapeNet dataset, annotated with part-level semantic labels. It consists of 16, 880 models, 2 − 6 parts for each category, and 50 part

3D Object Classification
Dataset. ScanObjectNN [50] collects a total of 15,000 scanned objects for 15 classes. This real-world dataset presents challenges to classification tasks due to inherent scan noise and occlusion. We benchmark Pix4Point on the hardest variant of this dataset, PB T50 RS.
Results. Tab. 3 shows the effectiveness of the proposed PViT in point cloud classification in the real-world dataset ScanOb-jectNN.
(1) PViT trained from scratch achieves better performance than the vanilla ST network as well as SSL-based methods Point-BERT [63] and Point-MAE [37].
(2) Image pretraining improves the overall accuracy (OA) and the mean accuracy (mAcc) of PViT by 2.2% and 3.2%, respectively. With Pix4Point image pretraining, PViT sets the new state of the art in ScanObjectNN classification with 87.9% OA.

Ablation Study and Analysis
In this section, we study the effects of pretraining strategies on the downstream point cloud tasks, including pretraining on different datasets (images or point clouds), using supervised or self-supervised pretraining. We further ablate the architecture design of the tokenizer and decoder for the Table 4. Effect of Pretraining Strategies. We show the downstream performance of PViT on S3DIS area 5 using the ViT-S backbone pretrained by (1) SSL on ShapeNet by Point-MAE [37] (2 nd row), (2) PViT supervised training from scratch through the ShapeNet part segmentation task (3 rd row), (3) SSL on ImageNet by MAE [20] (4 th row), and (4) supervised training on ImageNet by DeiT [49] (5 th row). The random initialized ST backbone (1 st row) is provided for reference. Network architectures (i.e. PViT) and the finetuning parameters are the same for all experiments for a fair comparison. Entire Network and Frozen Backbone refer to optimizing the whole network and only the tokenizer and decoder, respectively. Image pertaining improves point cloud understanding in various setups, especially when the ST backbone is frozen. proposed PViT network. Pretraining Strategies. Pix4Point pretrains the ST (ViT-S) using MAE on ImageNet-1K by default. Apart from this selfsupervised pretraining on images, there are multiple other ways to pretrain ST. It can be pretrained on 3D or 2D datasets, as well as, in a supervised or self-supervised manner. Here, we compare the following pretraining strategies: (1) selfsupervised pretraining on ImageNet using MAE [20]; (2) supervised pretraining on ImageNet using DeiT [49] without distillation; (3) self-supervised pretraining on ShapeNet using Point-MAE [37]; (4) supervised pretraining using the PViT network in the ShapeNet part segmentation task. Note that all the pretraining strategies share the same backbone from ViT-S, finetuning the same network (PViT) using the same finetuning parameters for a fair comparison. Tab. 4 compares the finetuning performance on S3DIS of the aforementioned training strategies. (1) While finetuning the entire network, the proposed image-pretrained ST reach mIoUs greater than 68 for both DeiT supervised pretraining (68.3 mIoU) and MAE self-supervised pretraining (69.3 mIoU), thus, outperforming the random initialized ST (64.4 mIoU) by more than 3.9 mIoU. (2) More interestingly, the image-pretrained ST also surpass the point cloud pretrained ST, both supervisely (PViT pretrained on ShapeNet part segmentation) and self-supervisely (Point-MAE) pretrained, by non-trivial margins (over 2 mIoU).

Conclusions and Future Work
This work presents a new network PViT and a framework Pix4Point to boost the performance of Standard Transformers in the task of point cloud understanding. PViT is an improved ST architecture for point cloud processing, using progressive point patch embedding for tokenizing point clouds and global representation appending to enrich global information. Pix4Point framework enables using publicly available images to pretrain the Transformer backbone and helps the downstream point cloud understanding. We conclude the following: (1) the proposed PViT network outperforms the vanilla ST model; (2) the image-pretrained Transformer can significantly improve performance in various 3D tasks; (3) ImageNet pretraining outperforms ShapeNet pretraining. We believe that our findings will benefit the community and motivate future work in this direction.
While the scope of this work is to investigate the benefits of using image-pretrained Transformers in the point cloud domain, a possible future work is to investigate a generalist vision model that is jointly optimized across the images and 3D point clouds. Such a generalist model might be helpful for a multi-modality scenario such as autonomous driving.

Improving Standard Transformer Models for 3D Point
Cloud Understanding with Image Pretraining -Supplementary Material -In this appendix, we provide additional content to complement the main manuscript: • Additional analysis on the effect of pretraining epochs; • Additional analysis on the effect of pretraining dataset size; • Detailed architecture configurations of PViT in various tasks.

A. Additional Analysis
We provide additional analysis on the pretraining epochs and pretraining dataset size in this supplementary material. We use the DeiT pretrained PViT without progressive encoder as the baseline. Pretraining Epochs. Here we study the effects of pretraining epochs on the downstream performance. We train DeiT without distillation using the entire training set of ImageNet-1K. We leverage the pretrained Transformers from different epochs and test their downstream performance in S3DIS area 5. Tab. I shows that the downstream performance drops at first mainly because the Transformer does not converge to good minima yet. The pretrained Transformer starts to outperform the random initialized Transformer (62.3 mIoU) at around epoch 50, gradually increases to reach peak performance (∼ 66 mIoU) at epoch 500, and begins to decline from epoch 600 mostly due to overfitting image data. Pretraining Dataset Size. Here, we ablate the size of the pretrained dataset by pretraining the ST using only a portion of ImageNet-1K. All models are trained using DeiT [49] without distillation by 300 epochs. Tab. II shows that the downstream performance in point cloud segmentation on S3DIS increases with the size of the pretraining dataset. The performance of Pix4Point continuously improves from 61.8 to 65.8. Increasing the size of the dataset from ImageNet-1K to ImageNet-21K (10 times larger) can further improve the performance by 0.8 mIoU.

B. Architecture
We list the detailed configuration of the proposed PViT architecture in Tab. III. The same Standard Transformer (12 layers, 6 heads, and 384 hidden dimensions) is used as the backbone in all tasks. The maxpooled feature among all tokens and the [CLS] are appended in the decoder to improve the segmentation and classification performance with global information. PViT uses a progressive tokenizer to subsample Table I. Effects of Pretraining Epochs on Downstream S3DIS Area 5 Segmentation. Pretraining strategy: DeIT without distillation on ImageNet-1K. PViT without a progressive encoder is used as the network. The pretrained Transformer reduces the downstream performance before epoch 50 due to that the backbone does not converge to a good minima yet, then outperforms the random initialized Transformer (61.8% mIoU), gradually increases the performance to reach a peak (∼ 66 mIoU) at epoch 500, and begins to decline the pretraining improvement from epoch 600 mostly due to overfitting image data.  the input point cloud to 1/16 with two downsampling stages in segmentation tasks. For classification, since the number of input points is small, PViT only downsamples once with a sampling ratio equal to 1/4.