{'fraction_non_alphanumeric': 0.03325220130288496, 'fraction_numerical': 0.018254706850884102, 'mean_word_length': 4.686749440260534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Auto-MVCNN: Neural Architecture Search for Multi-view 3D Shape Recognition

In 3D shape recognition, multi-view based methods leverage human's perspective to analyze 3D shapes and have achieved significant outcomes. Most existing research works in deep learning adopt handcrafted networks as backbones due to their high capacity of feature extraction, and also benefit from ImageNet pretraining. However, whether these network architectures are suitable for 3D analysis or not remains unclear. In this paper, we propose a neural architecture search method named Auto-MVCNN which is particularly designed for optimizing architecture in multi-view 3D shape recognition. Auto-MVCNN extends gradient-based frameworks to process multi-view images, by automatically searching the fusion cell to explore intrinsic correlation among view features. Moreover, we develop an end-to-end scheme to enhance retrieval performance through the trade-off parameter search. Extensive experimental results show that the searched architectures significantly outperform manually designed counterparts in various aspects, and our method achieves state-of-the-art performance at the same time.


Introduction
Along with the emergence of large 3D repositories [Chang et al., 2015;Wu et al., 2015] and the development of Convolution Neural Network (CNN), deep learning based 3D shape recognition has attracted strong interest in research [Su et al., 2015;Xie et al., 2017;Qi et al., 2017;Han et al., 2019]. Among different kinds of research works, multi-view based methods have achieved the best performance so far, in which images are generally first rendered from a set of views and then passed into CNNs to obtain a shape descriptor.
Handcrafted networks are usually adopted as the backbone in current methods, where a variety of classic architectures (e.g. VGG [Simonyan and Zisserman, 2015], ResNet [He et al., 2016]) have been employed for feature extraction. Over the past years, the majority of researches emphasized leveraging relationships among view images in single-view feature level [Ma et al., 2019;He et al., 2019] or multi-view feature level [Feng et al., 2018;Han et al., 2019], and therefore devoted efforts to designing sub-network on top of the backbone. Despite the remarkable progress achieved in previous studies, the effect of CNN extractors is not fully investigated, which restricts their performance to some extent. Meanwhile, in order to avoid the excessive memory usage, a lot of research works Ma et al., 2019;He et al., 2019] develop a multi-stage training scheme which only uses the backbone to extract view features, while the relation between the feature extraction and the feature fusion is neglected. These drawbacks may not only degrade the performance but also lead to being time-consuming and increasing computation cost. As the neural network plays a crucial role in 3D shape recognition, it is desired to design an esfficient and powerful architecture that can process multi-view images with an end-to-end scheme.
In recent years, due to the effectiveness of Neural Architecture Search (NAS) compared with the human-designed structures [Fang et al., 2020;Guo et al., 2020], its application field has also expanded on various benchmarks Real et al., 2019;Liu et al., 2019a;, before which NAS has been dedicated to image classification. Besides, it is worth mentioning that the combination of multiple loss functions is essential in multi-task learning, which has a large impact on neural architecture design. Unfortunately, many of AutoML methods using reinforcement learning and evolution algorithms have extreme computational demands. And there is a relatively small amount of works that study the balance of training as well as associated techniques in the searching process. Darts [Liu et al., 2019b] is a well-known gradient-based framework that largely reduces computation complexity. However, directly transferring Darts algorithm into 3D shape recognition is not an advisable choice. Firstly, Darts processes a single image instead of multi-view images, in which case the correlation across multiple views will be neglected. In addition, we aim to develop a unified model for both classification and retrieval tasks, which is different from Darts that focuses on single task optimization.
To automatically search a suitable neural network for 3D shape analysis, in this paper, we propose our Auto-MVCNN which is particularly adaptive for multi-view shape recognition. Our network architecture contains three parts: a shared backbone for view feature extraction, a fusion module for multi-view feature fusion and a linear combination of loss functions for multi-task learning. The pipeline of our method is shown in Fig. 1. In the network, we propose a novel fusion cell which is specially designed for processing sequence view features. By continuous relaxation of discrete operations, it inherits the efficiency and effectiveness of Darts and can be integrated into the existing framework seamlessly. The equipped search space enables us to find appropriate fusion patterns that explore the correlation among views. For supervision signals, in addition to the shape classification loss on the top, we also add a view classification loss function for view feature enhancement and another retrieval loss function for the retrieval task. The trade-off parameters that linearly combine these loss functions are searched by an end-to-end scheme.
To evaluate the performance of Auto-MVCNN, we carry out experiments on two large-scale datasets and conduct the comparison from various aspects. Compared to the handcrafted network, with or without ImageNet pre-training, our searched networks show the superiority in regard to both performance and computation resources saving. Besides, more experiments are implemented to compare our method with state-of-the-arts indicating the effectiveness of our proposed framework. Finally, we also analyze the impact of the number of initial channels and the stability of the searching process.
To summarize, the contribution of our paper is four-fold: 1. To our best knowledge, this is the first work of neural architecture search in the field of multi-view 3D shape recognition that replaces the manually designed search with the automatic mechanism. 2. We propose a novel fusion cell to process multi-view features that can be integrated into the existing framework seamlessly. 3. We develop a simple scheme that dynamically searches loss weights of multiple loss functions, achieving appropriate training balance for multi-task learning. 4. Extensive experiments show that the searched CNNs achieve state-of-the-art performance, using much fewer parameters than other baselines.
2 Related Work

Multi-view 3D Shape Recognition
On the basis of different formats of the processed 3D data, methods in 3D shape recognition could be roughly divided into two categories: model based methods [Osada et al., 2002;Xie et al., 2017;Li et al., 2020] and view based methods Wang et al., 2017a]. In this section, we mainly introduce multi-view based methods which leverage 2D views' information to construct 3D descriptors. MVCNN [Su et al., 2015] is a typical framework in which the whole pipeline is divided into two parts. The first part is the backbone for extracting view features and the other part is responsible for processing further shape features. Between the two parts, the view features are aggregated into a single shape representation through the element-wise maximum operation. In , a postprocessing algorithm adopting the inverted file is proposed for fast retrieval. Recently, leveraging correlation among views has become more and more popular in some research works. GVCNN [Feng et al., 2018] introduces a multi-level descriptor by exploring the view-group-shape hierarchical correlation, which largely improves the performance on 3D shape classification and retrieval. [Huang et al., ] develops a local 3D shape descriptor, which makes full use of relations over points on the shape and can be directly utilized for a wide range of shape analysis tasks. Many research works Feng et al., 2018;He et al., 2018] show that metric learning is essential in 3D shape retrieval task. [Li et al., 2019c] designs two loss functions to separately deal with these two distances. The flexible combination property of the proposed loss functions provides effective tools to enhance retrieval performance.

Neural Architecture Search
The basic ideology of NAS is to find candidate network structures through a search strategy in a defined search space, based on the obtained feedback of the evaluation. The search space develops from the entire structure at the beginning to stacking cells . Cell-based search can greatly narrow the search space and improve the search efficiency, which has been applied in numerous subsequent works. However, search strategies based on reinforcement learning, such as the Q-learning algorithm in MetaQNN [Baker et al., 2017], require high computational complexity. AmoebaNet [Real et al., 2019] develops evolutionary algorithms instead of reinforcement learning to optimize performance. Although it achieves better results, it still takes 450 GPUs and 7 days in a row to complete the experiment.
NAS is gradually approaching the very obvious problem of solving heavy computation, which enables gradient-based methods and other efficient methods to emerge. ENAS [Pham et al., 2018] employs weight sharing to accelerate validation, where the cell-based search mode greatly improves experimental results. Similar to ENAS, DARTS [Liu et al., 2019b] also searches subgraphs in designing cells and conducts weight sharing as well. EfficientNet  and MobileNetv3 [Howard et al., 2019] use the network search to obtain a fixed set of scaling factors to scale the width, depth, and resolution of the network respectively, achieving better efficiency and accuracy. To search for an appropriate loss function for face recognition, AM-LFS [Li et al., 2019a] employs the REINFORCE [Williams, 1992] idea to automatically search for appropriate hyperparameters of the loss function, with great transferability at the same time.

Auto-MVCNN Network
In our method, a view image sequence of length N v is input to a shared backbone to obtain view feature vectors F = [f 1 , f 2 , ..., f Nv ] ∈ R m×Nv . Then a shape descriptor d ∈ R n is generated by fusing F . The pipeline illustration is described in Fig. 1.
The whole neural network, which is called supernet, is formed of multiple cells and multi-task loss functions. It is divided into two parts according to their functions: the backbone is used to extract view features and the fusion module is designed for view feature fusion. The backbone consists of a number of normal cells inserted with several reduction cells which are presented in a stacking manner. Different from the backbone composition, the fusion module is generated by fusion cells. The details of these components will be described in Sec. 3.2 and Sec. 3.3.
In our method, both classification and retrieval tasks are fulfilled with multiple supervision signals and we treat all the loss functions as important roles in neural network architecture. Besides the classification loss L 1 on the top of the supernet, we add an auxiliary classification loss L 2 and an auxiliary retrieval loss L 3 to enhance view features and boost retrieval performance respectively. The formulation of the loss functions will be described in Sec. 3.4.

Backbone Search
In the searching stage, the cell is the basic searching component that contains the combination of all candidate operations. Formally, a cell C k is defined as a directed acyclic graph (DAG) which contains an ordered sequence of 7 nodes x (1) , x (2) , · · · , x (7) . Each node represents a latent tensor (i.e. a feature map) and each edge e i,j consists of multiple parallel network layers. Two input tensors x (1) and x (2) are the out-put of previous cells C k−1 , C k−2 and one output tensor x (7) is computed as (6) ). For the intermediate node, the computation can be formulated as whereō (i,j) is a continuous relaxation of the search space O: Here, α is the architecture parameter that represents the weight of operation o in the edge e i,j . The continuous relaxation allows us to optimize α by gradient descent. O is the set of candidate operations and we choose the same set as previous works [Liu et al., 2019b;Liu et al., 2019a] to keep consistence: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 atrous convolution, 3 × 3 average pooling,3 × 3 max pooling, skip connection, and zero.
A cell that remains the same spatial resolution as the previous cell is the normal cell and that divides spatial dimension by 2 is the reduction cell. In both training and inference stage, view features F are extracted simultaneously from the backbone.

Fusion Module Search
The NAS framework [Liu et al., 2019b] focuses on image classification that can generate single view feature. Though feasible, a simple combination (e.g. max-pooling, viewwise addition) of view-level features will lead to information loss that largely degrades the performance. By drawing on the experiences of previous works [Feng et al., 2018;  He et al., 2019], we summarize that leveraging the spatial information and the feature correlation among views is essential for obtaining competitive performance. Following the principle mentioned above, we design the fusion module in Auto-MVCNN to aggregate view features into a compact and discriminative shape feature. This module consists of two sequential fusion cells that are developed to process sequence view features F ∈ R m×Nv . In order to integrate the fusion cell into the existing optimization framework, the fusion cell has a search space O f which is similar to O that applies the size adaption to the kernel size of all operations. Concretely, we regard F as a three dimension tensor of shape m × N v × 1 with m channels and spatial dimension N v × 1. And the size adaption is to change the kernel size from k × k to k × 1 for all operations in O. In this way, we can adopt all the operations in O f on F . The size adoption of operations is illustrated in Fig. 2. The fusion cell is then formed by linking these operations using Eq. 2. It is worth pointing that the size adaption is equivalent to padding zeros to F such that its shape is m × N v × k, while the operations remain the same in O.
The compatibility of the fusion cell shows various excellent properties. For one operation o ∈ O f , o conserves the spatial relationship of input tensors and thus can model the spatial information among views. The correlation among view features can also be revealed by a variety of operations. In addition, the fusion cell inherits the diversity of the combination among different layers, which enables the fusion module to search for novel fusion patterns.

Trade-off of Loss Functions
Auto-MVCNN aims to develop a network for both shape classification task and shape retrieval task where training balance is extremely important. There are totally three loss functions in the supernet. L 1 and L 2 are softmax loss located on the top and middle of Auto-MVCNN, in charge of the shape classification and the view feature enhancement respectively. L 3 is the loss function proposed in [Li et al., 2019c] which is used for enlarging the inter-class distance: ( 3) where M and y are the batch size and the ground truth label respectively. In general, classification places emphasis on the right label prediction while for retrieval the feature distribution is more important. This phenomenon is also observed in Feng et al., 2018] and they adopt offline metric learning algorithms to improve retrieval performance. In this paper, by contrast, we tackle the issue by adding the loss function of metric learning L 3 with an end-to-end training scheme. In practice, multi-task loss functions are linearly combined, L = i ω i L i . Since our target is searching for proper training balance, we normalize the loss weight using trade-off parameters λ = (λ 1 , λ 2 , λ 3 ) and the total objective loss function is formulated as: An appropriate value distribution of λ could enhance performance without impeding the classification task, while a relative quick drop of loss i means its gradient is large in the backpropagation which hampers the training of other tasks. However, direct optimization via minimizing L total is infeasible. If L i is the minimum value among the three losses, λ i will be straight to become large, which is irrelevant to the balance of training. To tackle this issue, in this paper, we develop a scheme that searches for the balance of training leveraging the performance on the validation set. Specifically, let L i (t) denote the loss value in the t-th iteration on the validation set, and we define r i (t) = L i (t − 1)/L i (t) as the training rate of L i . We propose a regularization term that directly involves λ i and the training rate: L grad penalizes λ i when its corresponding loss drops quickly and, in turn, it augments the weight of one task if its training is relatively slow.

Optimization
In our method, searching neural network is a bilevel optimization problem that has two different sets of parameters: the network parameters W and the architecture parameters (α, λ). We follow the first-order approximation proposed in Darts [Liu et al., 2019b] and split the training data manually into two disjoint sets D train and D val . The optimization of W and (α, λ) is carried out in an alternating fashion on D train and D val until convergence, as shown in Alg. 1. The stability of search is clarified in Appendix. Algorithm 1: The Auto-MVCNN search algorithm Input: architecture parameters (α, λ), network parameters W , D train , D val while not converged do Sample mini-batch from D val , calculate L total (W, α, λ) and L grad (W, α, λ); Update α by descending ∇ α L total (W, α, λ); Update λ by descending ∇ λ L grad (W, α, λ); Sample mini-batch from D train , calculate L total (W, α, λ); Update W by descending ∇ W L total (W, α, λ); end

Evaluation
After the search convergence, the final cell for evaluation is pruned by selecting non-zero layers in the connection. The selection is achieved by retaining the top-k strongest operations from edge i to edge j: We use k = 2 in our method. The extracted cells are then stacked to form the supernet and retrained for evaluation. The final searched cells for evaluation are shown in Fig. 3  For the sake of stability, the loss weights need to rescaled to ensure the loss weight of L 1 is 1, i.e. (1, 0.95, 2.7), in the retraining. When we retrain the architecture, its initial number of channels changes to 24 and 36 (for matching the size of popular NAS architectures), generating our two representative models AM 24 and AM 36.

Dataset and Metrics
To evaluate the performance of our method, we conduct experiments on the Princeton ModelNet dataset [Wu et al., 2015] and the ShapeNetCore55 dataset [Savva et al., 2016b].
The ModelNet is a large-scale 3D shape dataset which contains 127,915 3D CAD models divided into 662 categories. We apply the extracted subset ModelNet40, which includes 12,311 models cleaned manually from 40 categories, in our evaluation. We follow the same training/testing split as described in [Su et al., 2015], by randomly selecting 100 unique models per category from the subset, where 80 models are used for training and the rest for testing. The evaluation metrics adopted in this dataset include the (per-class) classification accuracy, the mean average precision (mAP) and the area under curve (AUC). Their detailed definitions can be found in [Wu et al., 2015].
The ShapeNetCore55 dataset, introduced in the Shape Retrieval Contest (SHREC) 2016 competition track, contains 51,190 3D shapes from 55 common categories which is a subset of the full ShapeNet dataset with clean 3D models. Each model in this dataset is attached with a label from the 55 categories plus a fine-grained subcategory deriving from 204 subcategories. The dataset is divided into two versions, named as "normal" version and "perturbed" version, where the 3D shapes in the former version are aligned but more challenging in the latter one with all shapes are rotated randomly. In terms of training and testing split method, 70% shapes in the dataset are provided for training and another 10% shapes are for validation, with the remaining 20% shapes forming the testing set. Refer to [Savva et al., 2016b] for the definition of metrics F-Measure (F-1) and NDCG used in this paper.

Implementation Details
The experiments are carried out on a server with four Nvidia GTX2080Ti GPUs, Intel Xeon CPU E5-2678 v3 and 128G RAM. Before training and testing, each shape is rendered to generate 12 images with size 224×224, following the same protocol as . Architecture search on ModelNet40. In the experimental settings, we employ 3 normal cells, 2 reduction cells and 2 fusion cells to build the architecture space. The supernet contains a stem at the bottom with 7 cells stacked sequentially, Fig. 1 displays the architecture. During the searching process, half of the ModelNet40 training data is set as the validation set D val . The batch size is 36 and the initial number of channels is 16. Please refer to Appendix for other hyperparameter settings. Architecture evaluation on ModelNet40 and ShapeNet-Core55. To evaluate the performance of searched architectures, we need to retrain the derived supernet on our target dataset. The retraining set the batch size to 36. SGD with the initial learning rate 0.01 is adopted. The shape descriptor d is extracted for the retrieval task using cosine distance. For a fair comparison with other methods, we also pretrain the supernet on ImageNet classification benchmark. We propose two models, AM c24 and AM c36, with initial channels 24 and 36 respectively for evaluation. Please refer to Appendix for detailed hyperparameter settings.

Main Results
Comparison with hand-crafted networks. As for comparative experiments, we train several popular hand-crafted networks in this domain using the same training protocol. The comparative experiments involve several aspects, results of which are indicated in Tab. 1. We employ the network parameters (Params) and the multiply-accumulate operation (MACs) to measure the network size and the computation cost. Cputime is the network inference time averaged by 10 times running. Their values are obtained by inputting an image sequence (12 × 3 × 224 × 224) into the network. For a comprehensive comparison, we take the following factors into consideration: (1) the position of the fusion layer,   AM c36 achieves better performance compared to AM c24. Our proposed Auto-MVCNN is also evaluated on ShapeNetCore55 perturbed dataset. This perturbed dataset is more challenging as all shapes are rotated randomly. Note that the architecture is also searched on the ModelNet40. We choose the participants of the competition [Savva et al., 2016a;Klokov and Lempitsky, 2017] and other popular methods as comparison. As is shown in Tab. 3, our method (AM c36) outperforms others in both mAP and NDCG metrics. We attribute the little lower performance on F-Measure to the transfer of datasets.

Ablation Study
Effect of fusion module. To demonstrate the effectiveness of our fusion cells, in our searched network, we manually replace the fusion cells with normal cells and conduct experiments on ModelNet40. Note that we maintain the same number of layers and same supervision signals. We also choose three popular cell-based NAS networks that have similar network size to ours as comparison (the supervision is single softmax loss). As these architectures are searched for single image classification, we adopt a view max-pooling operation to fuse the view features after the penultimate layer as [Su et al., 2015]. As we can see from Tab. 4, owing to the ability of the fusion module that can explore the intrinsic correlation among view features, our learned network outperforms other NAS architectures. When compared with [Howard et al., 2019] and , the following two other factors also contribute to performance improvement: (1) Our network is derived directly from the ModelNet40 dataset while others are searched on classification benchmarks; (2) Multiple supervision signals and corresponding appropriate loss weights enhance the performance on both shape classification and shape retrieval. Effect of dynamic loss balance. To reveal the superiority of our loss weights balance method, we choose several commonly adopted loss combinations and compare their performances. The results are shown in Tab. 5. Note that values in the loss combinations need to be rescaled to conduct retraining (see Sec. 5.2). We can see from the first three experiments that both L 2 and L 3 are essential for competitive retrieval performance and our result is better than others. When the loss combination is close to ours, its corresponding performance   is also similar to ours.

Conclusions
In this paper, aiming at the problem of multi-view 3D shape recognition, we propose a novel neural architecture search framework to optimize architectures, which is named as Auto-MVCNN. It abandons hand-crafted networks as the backbone for the first time, which greatly reduces the number of parameters and computational complexity. The proposed fusion cell enables the whole network to explore the intrinsic connections of view features automatically, which fully utilizes the 3D information. In addition, we apply a searching scheme for the training balance with an end-to-end fashion, improving both classification and retrieval performances. Extensive experiments exhibit our Auto-MVCNN achieves the best performance in various aspects, and clarify its effectiveness at the same time. We adopt the stochastic gradient descent (SGD) with the initial learning rate 0.01, the momentum 0.9 and the weight decay 3e-4 to optimize the network weights W . The architecture parameter (α, λ) is initialized by a Gaussian distribution of the mean value 0 and the standard deviation 1e-3. λ is optimized using the same optimizer of W and the learning rate is 0.05. α is optimized by Adam [Kingma and Ba, 2015] with the initial learning rate 3e-4, the momentum (0.5, 0.999) and the weight decay 1e-3. For the sake of stability, the gradient clip is adopted and a warmup scheme is conducted during the searching process.

A.2 Pretrain on ImageNet
In 3D shape recognition, most approaches take advantage of ImageNet pretrained network to boost their performances. For a fair comparison, we also train the searched network on the ImageNet classification benchmark. What should be noticed is that we train separate view images for the classification task, therefore only the parameters of the first 5 cells are updated. The training process takes 800 as the batch size and total epochs are 120. SGD with the initial learning rate 0.1 and weight decay 3e-4 is used in the optimization.
A.3 Architecture evaluation on ModelNet40 and ShapeNetCore55.
To evaluate the performance of searched architectures, we need to retrain the derived supernet on our target dataset. The retraining set the batch size to 36. SGD with the initial learning rate 0.01 is adopted. The weight decay is 1e-3 without pretraining and 3e-4 with pretraining. The shape descriptor d with dimension init channels × 16 is used to conduct retrieval.

A.4 Stability of search.
Since the searching process is initialization-sensitive, the final searched architectures are generally distinct from one another due todifferent random seeds. To investigate the performance stability of the searching process, we conduct the search experiment 5 times with the same hyperparameters but different random seeds. The results of the experiments are shown in Tab. 6.