{'fraction_non_alphanumeric': 0.041192475351803634, 'fraction_numerical': 0.01954302097656593, 'mean_word_length': 4.994738688179586, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PointCLIP: Point Cloud Understanding by CLIP

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.


Introduction
Deep learning has dominated computer vision tasks of both 2D and 3D domains in recent years, such as image classification [12,17,21,28,36,41], object detection [1, 4, * Equal contribution. † Corresponding author.  Figure 1. A Comparison of Training-testing schemes between PointCLIP and PointNet++. Different from classical 3D networks, our proposed PointCLIP is pre-trained by 2D image-text pairs, but conducts zero-shot classification on 3D datasets, which achieves cross-modality knowledge transfer. 13,29,46,64], semantic segmentation [3,24,35,61,65], point cloud recognition and part segmentation [19,43,44,55]. With 3D sensing technology developing rapidly, the growing demand for processing 3D point cloud data has boosted many advanced deep models with better local feature aggregator [30,32,49], geometry modeling [20,39,40] and projection-based processing [20,34,48]. Different from grid-based 2D image data, 3D point clouds suffer from space sparsity and irregular distribution, which hinder direct methods transfer from 2D domain. Additionally, large-scale newly captured point cloud data contain a large number of objects of "unseen" categories to the trained classifier. In this scenario, even the best-performing models might fail to recognize them and it is unaffordable to re-train every time when "unseen" objects arise. Similar issues have been dramatically mitigated in 2D vision by Contrastive Vision-Language Pre-training (CLIP) [45], which proposed to learn transferable visual features with natural language supervisions. For zero-shot classification of "unseen" categories, CLIP utilizes the pretrained correlation between vision and language to conduct open-vocabulary recognition and achieves promising performance. To further enhance the accuracy in few-shot settings, CoOp [66] adopted learnable tokens to encode the text prompts, so that the classifier weights can be adaptively formed. From another perspective, CLIP-Adapter [16] appends a lightweight residual-style adapter with two linear layers for better adapting image features. Tip-Adapter [63] further boosts its performance while greatly reduces the training time. Both methods achieve significant improvements over zero-shot CLIP. Consequently, the problem of recognizing new unlabeled objects has been explored by CLIP in 2D. However, a question is naturally arised: Could such CLIP-based models be transferred to 3D domain and realize zero-shot classification for "unseen" 3D objects?
To address this issue, we propose PointCLIP, which transfers CLIP's 2D pre-trained knowledge to 3D point cloud understanding. The first concern is to bridge the modal gap between unordered point clouds and the gridbased images that CLIP could process. Considering the need for real-time prediction in various scenarios, such as autonomous driving [4,13,29,42] and indoor navigation [67], we propose to adopt online perspective projection [19] without any post rendering [48], i.e., simply projecting each point onto a series of pre-defined image planes to generate scatter depth maps. The cost of this projection process is marginal in both time and computation, but reserves the original property of the point cloud from multiple views. On top of that, we apply CLIP to encode multiview features of point cloud by the CLIP pre-trained visual encoder and obtain each view's text-matched prediction independently via zero-shot classifier. Following CLIP, we place 3D category names into a hand-crafted template as prompts and generate the zero-shot classifier by CLIP's textual encoder. As different views contribute differently to the recognition of entire scene, we obtain the final prediction for point cloud by weighted aggregation between views. Although PointCLIP achieves cross-modality zero-shot classification without any 3D training, its performance still falls behind classical point cloud networks well-trained on full datasets. To eliminate this gap, we introduce a learnable inter-view adapter with bottleneck linear layers to better extract features from multiple views in few-shot settings. Specifically, we concatenate all views' features and extract the compact global feature of the point cloud via interacting and summarizing cross-view information. Based on the global representation, adapted feature of each view is generated and added to their original CLIP-encoded feature via a residual connection. In this way, each view is equipped with the fused global feature and also combines newly adapted feature from the 3D few-shot dataset with 2D pre-trained CLIP's encoding. During training, we only fine-tune this lightweight adapter and freeze CLIP's both visual and textual encoders to avoid over-fitting, since only a few samples per class are given. Surprisingly, PointCLIP with an interview adapter with few-shot fine-tuning achieves comparable performance with some previous models well-trained with full datasets, which is a good trade-off between performance and cost.
Additionally, we observe that CLIP's 2D knowledge, supervised by contrastive loss, is complementary to the closeset 3D supervisions. The PointCLIP with an inter-view adapter can be fine-tuned under few-shot settings to improve the performance of classical fully-trained 3D networks. Taking PointCLIP in 16-shot ModelNet40 [57] and fully-trained PointNet++ [44] as an example, we directly ensemble their predicted logits for testing. Surprisingly, the performance of PointNet++'s 89.71%, is enhanced to 92.03% by PointCLIP with an accuracy of 87.20%. Furthermore, we select CurveNet [39], the state-of-the-art 3D recognition model, as the ensembling baseline, and achieve performance boost from 93.84% to 94.08%. In contrast, simply ensembling two models fully trained on ModelNet40 without PointCLIP only leads to performance loss. Therefore, PointCLIP could be regraded as a multiknowledge ensembling module, which promotes 3D networks via 2D contrastive knowledge with limited additional training.
The contributions of our paper are as follows: • We propose PointCLIP to extend CLIP for handling 3D point cloud data, which achieves cross-modality zero-shot recognition by transferring 2D pre-trained knowledge into 3D.
• An inter-view adapter is introduced upon PointCLIP via feature interaction among multiple views and improves the performance of few-shot fine-tuning.
• PointCLIP can be utilized as a multi-knowledge ensembling module for enhancing performance of existing fully-trained 3D networks, which surpasses stateof-the-art performances.
• Comprehensive experiments are conducted on widely adapted ModelNet10, ModelNet40 and the challenging ScanObjectNN, which indicate PointCLIP's potential for 3D understanding.

Related Work
Zero-shot Learning in 3D. The objective of zero-shot learning is to enable recognition of "unseen" objects which are not adopted during training. Although zero-shot learning has drown much attention on 2D classification [26,45,58], only a few works explore how to conduct it in 3D domain. As the first attempt on point cloud, [7] divides the 3D dataset into two parts: "seen" and "unseen" samples, and trains PointNet [43] on the former but tests on the latter by measuring cosine similarities with category semantics. Based on this prior work, [5] further mitigates the hubness problem [62] resulted from low-quality extracted 3D features and [6] introduces a triplet loss for better performance in transductive settings, which allows to utilize unlabeled "unseen" data at training time. Different from all above settings, which train the network on part of the 3D samples and predict on the others, PointCLIP achieves direct zeroshot recognition without any 3D training and conducts prediction on the whole point cloud datasets. Thus, our setting is more challenging for the domain gap between 2D pretraining and 3D application, but more urgent for practical problems.
Transfer Learning. Transfer learning [9,60] aims to utilize the knowledge from data-abundant domains to help the learning on data-scarce domains. For general vision, Ima-geNet [9] pre-training can greatly assist downstream tasks, such as object detection [1,18,46] and semantic segmentation [35]. Also in natural language processing, representations pre-trained on web-crawled corpus via Mask Language Model [10] achieves leading performance on machine translation [38] and natural language inference [8].
Without any fine-tuning, the recently introduced CLIP [45] shows superior image understanding ability for "unseen" datasets. CLIP-Adapter [16], Tip-Adapter [63], Action-CLIP [53] and WiSE-FT [56] further indicate that the performance of CLIP can be largely improved by infusing domain-specific supervisions. Although the successes stories are encouraging, most of the existing methods conduct knowledge transfer within the same modality, namely, image to image [9], video to video [2] or language to language [10]. Different from them, our PointCLIP is able to efficiently transfer representations learned from 2D images to the disparate 3D point clouds, which motivates future researches on transfer learning across different modalities.
Deep Neural Networks for Point Cloud. Existing deep neural networks for point cloud can be divided into pointbased and projection-based methods. Point-based models process on raw points without any pre-transformation. PointNet [43] and PointNet++ [44] firstly encode each point with a Multi-layer Perceptron (MLP) and utilize max pooling operation to realize permutation invariance. Recent point-based methods propose more advanced local aggregators and architecture designs [30,49]. Other than raw points, projection-based methods understand point cloud by transferring it to volumetric [37] or multi-view [48] data forms. Therein, multi-view methods project point cloud into images of multiple views and process them with 2D Convolution Neural Networks (CNN) [21] pre-trained on Ima-geNet [28], such as MVCNN [48] and others [14,15,25,59]. Normally, such view-projected methods operate on offlinegenerated images which are projected from point-converted 3D meshes [54] or required post-rendering [47] for shades and textures, so they are costly and impractical to be adopted for real-time applications. On the contrary, we follow SimpleView [19], to naively project raw points onto image planes and set their pixel values according to the vertical distance. Such depth-map generation results in marginal time and computation costs, which meets the demand for efficient end-to-end zero-shot recognition.

Method
In Section 3.1, we first revisit Contrastive Vision-Language Pre-training (CLIP) for 2D zero-shot classification. Then in Section 3.2, we introduce our PointCLIP, which transfers 2D pre-trained knowledge into 3D. In Section 3.3, we provide PointCLIP with inter-view adapter for better performance under few-shot settings. In Section 3.4, we propose to ensemble PointCLIP with fully-trained classic 3D networks for multi-knowledge ensembling, which can achieve state-of-the-art performance.

A Revisit of CLIP
CLIP is trained to match images with their corresponding natural language descriptions. There are two independent encoders in CLIP, respectively for visual and textual features encoding. During training, given a batch of images and texts, CLIP extracts their features and learns to align them in the embedding space with a contrastive loss. To ensure comprehensive learning, 400 million training imagetext pairs are collected from the internet, which enables CLIP to align images with any semantic concepts in an open vocabulary for zero-shot classification.
Specifically, for an "unseen" dataset of K classes, CLIP constructs the textual inputs by placing all category names into a pre-defined template, known as prompt. Then, the zero-shot classifier, denoted as W t ∈ R K×C , is obtained by the C-dimensional textual feature of category prompts. Each of the K row vectors in W t encodes the pre-trained category weights. Meanwhile, the feature of every test image is encoded by CLIP's visual encoder to f v ∈ R 1×C and the classification logits ∈ R 1×K are computed as, where softmax i (·) and p i denote the softmax function and predicted probability for category i. The whole process does not require new training images, but achieves promising zero-shot classification performance only by frozen pretrained encoders.

Point Cloud Understanding by CLIP
A variety of large-scale datesets [28,31] in 2D provide abundant samples to pre-train models [11,21] for highquality and robust 2D features extraction. In contrast, the widely-adopted 3D datasets are comparatively much smaller and have limited categories, e.g. ModelNet40 [57] with 9,843 samples and 40 classes vs. ImageNet [28] with 1 million samples and 1,000 classes. Thus, it is very difficult to obtain good pre-trained 3D networks for transfer learning. To alleviate this problem and explore the crossmodality power of CLIP, we propose PointCLIP to conduct zero-shot learning on point clouds based on the pre-trained CLIP.
Bridging the Modal Gap. Point cloud is a set of unordered points scattering in the 3D space, its sparsity and distribution greatly differ from grid-based 2D images. To convert point clouds into CLIP-accessible representations, we generate point-projected images from multiple views to eliminate the modal gap between 3D and 2D. In detail, if the coordinate of a point is denoted as (x, y, z) in the 3D space, taking the bottom projection view as an example, its location on the image plane is ( x/z , y/z ) following [19]. In this way, the projected point cloud is a foreshortened figure, namely, small in the distance but big on the contrary, which is more similar to that in real photos. Other than [19] applying convolution layers to pre-processing the one-channel depth map into three, we do not adopt any preconvolution and directly set the pixel value equaling to z in all three channels. Also, different from other off-line projection methods, whose projected images are generated from meshes [54] or CAD models [48], our projected depth maps are from raw points and contain no color information but scattered depth values, which leads to marginal time and computation cost. With this lightweight cross-modality cohesion, CLIP's pre-trained knowledge can be then utilized for point cloud understanding.
Zero-shot Classification. Based on projected images from M views, we use CLIP to extract their visual features {f i }, for i = 1, . . . , M . For the textual branch, we place K category names to the class token position of a predefined template: "point cloud depth map of a [CLASS]." and encode their textual features as the zero-shot classifier W t ∈ R K×C . On top of that, classification logits i of each view are separately calculated and the final logits p of point cloud are acquired by their weighted summation, where α i is a hyper-parameter weighing the importance of view i. Each view f i encodes a different perspective of the point cloud feature, which is capable for independent zero-shot classification. Their summation further complements the information of different perspectives to obtain an overall understanding. The whole process of PointCLIP is non-parametric for the "unseen" 3D dataset, which pairs each point cloud with its category via CLIP's pre-trained 2D knowledge and without any 3D training.

Inter-view Adapter for PointCLIP
Although PointCLIP achieves efficient zero-shot classification on point clouds, its performance is still incomparable to fully-trained 3D neural networks [43,44]. We then consider a more common scenario where a few objects of each "unseen" category are contained in the newly collected data, and networks are required to recognize them under

Residual Connection
Inter-view Adapter   such few-shot settings. It is impractical to fine-tune the whole model, since the enormous parameters and insufficient samples would easily result in over-fitting. Therefore, referring to [23] in Natural Language Processing (NLP) and CLIP-Adapter [16] for fine-tuning pre-trained models on downstream tasks, we append a three-layer Multi-layer Perceptron (MLP) on top of PointCLIP, named inter-view adapter, to further enhance its performance under few-shot settings. For training, we freeze CLIP's both visual and textual encoders and fine-tune the learnable adapter via a crossentropy loss.
To be specific, given CLIP-encoded M -view features of a point cloud, we concatenate them along the channel dimension as Concate(f 1∼M ) ∈ R 1×M C , and acquire the compact global feature of point cloud via the first two layers of the inter-view adapter as where f global ∈ R 1×C and W 1 , W 2 stand for two-layer weights in the adapter. By this inter-view aggregation, features from multiple perspectives fuse into a summative representation. After that, the view-wise adapted feature is generated from the global feature and added to its original CLIP-encoded feature via a residual connection as where W 3i ∈ R C×C denotes the i-th part of W 3 for view i, and W T 3 = [W T 31 ; W T 32 ; · · · W T 3M ] ∈ R C×M C . On the one hand, f a i blends global-guided adapted feature into f i for the overall understanding of the point cloud and, thus, better view-wise prediction. On the other hand, the residualstyle adapter infuses newly-learned 3D few-shot knowledge with that of 2D pre-trained CLIP, which further promotes the cross-modality knowledge transfer.
After the inter-view adapter, each view conducts classification with the adapted feature f a i and the textual classifier W t . Same as zero-shot classification, all M logits from all views are summarized to construct the final prediction, and the view weights α i can be learnable parameters here for more adaptive aggregation. Surprisingly, just fine-tuning this lightweight adapter with few-shot samples contributes to significant performance improvement, e.g. from 20.18% to 87.20% on ModelNet40 with 16 samples per category, less than 1/10 of the full data. This inspirational boost demonstrates the effectiveness and importance of feature adaption on 3D few-shot data, which greatly facilitates knowledge transfer from 2D to 3D. Consequently, PointCLIP with inter-view adapter provides a promising alternative solution for point cloud understanding. In some applications, there is no condition to train the entire model with large-scale fully annotated data, and fine-tuning only the three-layer adapter with few-shot data can achieve comparable performance.

Multi-knowledge Ensembling
Classical point cloud networks, such as the early Point-Net [43] and the recent CurveNet [39], are trained from scratch on 3D datasets by close-set supervision. In contrast, PointCLIP mostly inherits pre-trained priors from 2D vision-language learning, containing different aspects of knowledge. We then investigate if the two forms of knowledge can be ensembled together for joint inference. In practice, we first obtain the classical model, e.g. Point-Net++ [44] pre-trained from [22], and PointCLIP of either zero-shot or the adapter version. We conduct inferences of the two models and ensemble their predicted logits by simple addition as the final output. Beyond our expectation, aided by 16-shot fine-tuned PointCLIP of 87.20%, PointNet++ of 89.71% is enhanced to 92.03% with a significant improvement of +2.32%. In other words, ensembling of two low-score models can produce a much stronger one, which fully demonstrates the complimentary interaction of knowledge from the two models. Also, even with the zero-shot PointCLIP of 20.18%, PointNet++ can still be improved to 92.10%. In contrast, ensembling a pair of classical full-trained models would not enhance the performance, which indicates the importance of complimentary knowledge. We also implement this ensembling with other advanced networks and observe similar performance boosts, some of which achieve state-of-the-art performances. Therefore, PointCLIP can be utilized as a plugand-play enhancement module to achieve robust point cloud understanding.

Zero-shot Classification
Settings. We evaluate the zero-shot classification performance of PointCLIP on three well-known datasets: Mod-elNet10 [57], ModelNet40 [57] and ScanObjectNN [51]. For each dataset, we require no training data and adopt the full test set for evaluation. For the pre-trained CLIP model, we adopt ResNet-50 [21] as the visual encoder and transformer [52] as the textual encoder by default. We then project the point cloud from 6 orthogonal views: front, right, back, left, top and bottom, and each view has a relative weight value ranged from 1 to 10, shown in the fourth column of  nal classification with different importance, but for ScanOb-jectNN which contains noisy points of floors and ceilings, the top and bottom views could hardly provide any information.
Ablations. In Table 2, We conduct ablation studies of zero-shot PointCLIP concerning projection view numbers and the importance of each view on ModelNet40. For the number of projected views, we try 1, 4, 6, 8, 10 and 12 1 views, for increasingly capturing the multi-view information of point clouds, but more than 6 views would bring redundancy and lead to performance decay. To explore how different views impact the performance, we unify all relative weights to 3 and respectively increase each view's weight to 9. As is shown in the table, projection from the right achieves the highest performance, which indicates its leading role, and the top and down views contribute relatively less to the zero-shot classification. In Table 4, we implement different visual backbones from ResNet [21] to vision transformer [11], and RN50×16 [45] achieves the best performance of 23.78%, which has 16 times more computations than ResNet-50. However, upgrading ResNet-50 to ResNet-101 with more parameters and deeper layers would not provide higher classification accuracy.
Prompt Design. We present five prompt designs for zeroshot PointCLIP in Table 3. We observe that the naive "a photo of a [CLASS]." achieves 17.02% on ModelNet40, but simply inserting the word "point cloud" into it would hurt the performance. We then remove "a photo" and directly utilize "point cloud" as the subject, which benefits the accuracy by +1.66%. Also, as the projected point cloud normally covers most of the image area, appending an adjective "big" could bring further performance improvement. Furthermore, we add the "depth map" to describe the projected images more relevantly, which contributes to the 1 The settings of views are in the Appendix.
best-performing 20.18%, demonstrating the importance of prompt choices.

Few-shot Classification
Settings. We experiment PointCLIP with the inter-view adapter under 1, 2, 4, 8, 16 shots also in the three datasets: ModelNet10 [57], ModelNet40 [57] and ScanOb-jectNN [51]. For K-shot settings, we randomly sample K point clouds from each category of the training set. We inherit the best projection settings from zero-shot experiments in Section 4.1. In contrast, considering both efficiency and performance, we adopt ResNet-101 [21] as CLIP's pretrained visual encoder for stronger feature extraction, and increase the projected view numbers to 10, adding the views of upper/bottom-front/back-left corners, since the left view is proven to be the most informative for few-shot recognition in Table 2. In addition, we modify the prompt to "point cloud of a big [CLASS].", which performs better in the fewshot experiments. For the inter-view adapter, we construct a residual-style Multi-layer Perceptron (MLP) consisting of three linear layers, as described in Section 3.3.
Performance. In Figure 5, we present the few-shot performances of PointCLIP and compare it with 4 representative 3D networks: PointNet [43], PointNet++ [44], Simple-View [19] and the state-of-the-art CurveNet [39]. As we can see, PointCLIP with inter-view adapter surpasses all other methods for the few-shot classification. When there are only a small number of samples per category, PointCLIP has distinct advantages, exceeding PointNet by 25.49% and CurveNet by 12.29% on ModelNet40 with 1 shot. When given more training samples, PointCLIP still leads the performance, but the gap becomes smaller due to the limited fitting capacity of the lightweight three-layer adapter. For the detailed training settings, please refer to the Appendix.
Ablations. In Table 2 contributes on ModelNet40. Differing from the zero-shot version, 10 views of 16-shot PointCLIP performs better than 6 views, probably because the newly-added adapter is able to better utilize the information from more views and adaptively aggregate them. For the importance of views, we follow the configurations of our zero-shot version and observe the reversed conclusion that, the left view is the most informative here. Surprisingly, for different visual encoders in Table 4, ResNet-101 achieves the highest accuracy with less parameters than vision transformer or ResNet-50×16. Table 3 lists the performance influence caused by prompt designs, and the "point cloud of a big [CLASS]." performs the best, which is slightly different from the analysis in Paragraph 4.1.

Multi-knowledge Ensembling
Settings. To verify the complementarity of blending pretrained 2D priors with 3D knowledge, we aggregate the fine-tuned 16-shot PointCLIP of 87.20% on Model-Net40, respectively with fully-trained PointNet [43], Point-Net++ [44], DGCNN [55], SimpleView [19] and Cur-veNet [39], whose trained models are obtained from [22,50] without any voting. We manually modulate the fusion ratio between PointCLIP and each model, and report the performance with the best Ratio in Table 5, which represents PointCLIP's relative weight to the whole.
Performance. As shown in Table 5, ensembling with PointCLIP improves the performances of all classical fullytrained 3D networks. The results fully demonstrate the complementarity of PointCLIP to existing fully-trained 3D models, and the performance gain is not simply achieved by ensembling models. These are surprising results to us, because the accuracy of 16-shot PointCLIP is lower than all other models trained with full datasets, but could still benefit their already high performances to be higher. Therein, the largest accuracy improvement is on Point-Net++ from 89.71% to 92.10%, and combining PointCLIP with the state-of-the-art CurveNet further achieves 94.08%. Also, we observe that, for models with low baseline per- formances, PointCLIP's logits need to account for a large proportion, but for the well-performing ones, such as Cur-veNet, their knowledge is supposed to play a dominant role in the ensembling.
Ablations. We conduct ablation studies of ensembling two models fully trained on ModelNet40 without Point-CLIP, and fuse their logits with the same ratio for simplicity. As is presented in Table 6, ensembling PointNet++ lowers the performance of RSCNN and CurveNet, and aggregating the highest two models, SimpleView and CurveNet, could not achieve better performance. Also, a pair of Point-CLIP would hurt the performance. Hence, simply ensembling two models with the same training scheme normally leads to performance degradation, which demonstrates the significance of multi-knowledge interaction. In Table 7, we fuse zero-shot PointCLIP and the model fine-tuned by 8, 16, 32, 64 and 128 shots, respectively with CurveNet to explore their ensembling performances. As reported, zeroshot PointCLIP with only 20.18% could enhance CurveNet by +0.04%. However, too much training on 3D dataset would adversely influence the ensembling accuracy. This is possibly caused by the too high similarity between two models, which cannot provide complementary knowledge as expected.

Conclusion and Limitation
We propose PointCLIP, which conducts cross-modality zero-shot recognition on point cloud without any 3D training. Via multi-view projection, PointCLIP efficiently transfers CLIP's pre-trained 2D knowledge into the 3D domain. Under few-shot settings, we design a lightweight inter-view adapter to aggregate multi-view representations and generate adapted features. By fine-tuning such adapter and freezing all other modules, the performance of PointCLIP is largely improved. In addition, PointCLIP could serve as a plug-and-play module to provide complimentary information for the classical 3D networks, which surpasses state-of-the-art performance. Although PointCLIP realizes the transfer learning from 2D to 3D, how to utilize CLIP's knowledge for other 3D tasks is still under explored. Our future work will focus on generalizing CLIP for wider 3D applications.
For ablation studies of projected view numbers, we adopt different settings for zero-shot and few-shot PointCLIP. As the right view is the most important for zero-shot Point-CLIP, we set the 12 views to: front, right, back, left, top, bottom, upper/lower right diagonal front/back (4 views) and upper left diagonal front/back (2 views). In contrast, few-shot PointCLIP achieves higher performance with left views, so we replace all the "left" settings above into "right". For both versions, the view number of M represents picking the first M views for experiments.
For PointCLIP with inter-view adapter, we fine-tune it under 1, 2, 4, 8 and 16 shots with batch size 32 and learning rate 0.01 for 250 epochs. Stochastic Gradient Decent (SGD) [27] with momentum 0.9 is adopted as the optimizer. We utilize a cosine scheduler for learning rate decay and Smooth Loss [55] following [19]. In Mod-elNet10 and ModelNet40, We apply random scaling and translation for training augmentation, but in the challenging ScanObjectNN, we append jitter and random rotation following [43]. During training, we freeze CLIP's both visual and textual encoders, and only fine-tune the inter-view adapter. For other compared models, we unfreeze all the parameters, and adopt the same data augmentation and loss functions reported in the papers.

C. Supplementary Ablations
Inter-view Adapter. We adopt the inter-view adapter with three linear layers: one for global extraction and two for view-wise adapted features generation. Here, we explore other architectures of the adapter on 16-shot Point-CLIP for ModelNet40 in Table 8. Specifically, w/o global denotes the adapter processing each view separately without interaction, and the w/o view-wise version repeats the global feature as each view's adapted feature. The 2-layer adapter removes the linear layer after the global representation and the pre-layer version moves it before the global extraction. The results show that dropping or changing the original modules in the adapter would all hurt the performance, especially the inter-view extraction of global feature. Adapted Features Fusion. The view-wise adapted feature is generated by the adapter and then added to the original CLIP-encoded feature via a residual connection. On ModelNet40, we evaluate the performance of 16-shot Point-CLIP with different fusion ratios β, which denotes the proportion of adapted features. To show the effect of β, we set all view weights the same. From the results in Table 9, different ratios actually lead to little performance variance and the β of 0.6 perfoms better than others. Thus, we adopt 0.6 as the fusion ratio by default, which indicates the comparable contributions between 2D pre-trained knowledge and 3D learned knowledge.  Table 9. Different fusion ratios of adapted features.

Architectures of Inter-view
Full Training Set. We also fine-tune PointCLIP on full training set of ModelNet40 [57] and present the results in Table 10. Likewise, we freeze both pre-trained visual and textual encoders in CLIP and only train the inter-view adapter. As expected, visual encoders with more parameters lead to higher accuracy, and only fine-tuning the appended lightweight adapter could achieve the performance of 92.01%.  Table 11. Ablations of PointCLIP fine-tuning different modules. denotes fine-tuning the module and symbol -denotes freezing.

D. Visualizations
We visualize some cases of ensembling PointCLIP with PointNet++ [44] to reveal the effectiveness of enhancement. As shown in Figure 6, two models both predict correctly for the four samples, and the ensembled model preserves the prediction. As for samples in the second and the third rows, PointCLIP and PointNet++ show the complementary properties that the ensembled model would rectify one of their wrong predictions, which demonstrates the importance of knowledge interaction.