{'fraction_non_alphanumeric': 0.03437401450646484, 'fraction_numerical': 0.015373699148533586, 'mean_word_length': 4.890178778732296, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor

Weight sharing based and predictor based methods are two major types of fast neural architecture search methods. In this paper, we propose to jointly use weight sharing and predictor in a unified framework. First, we construct a SuperNet in a weight-sharing way and probabilisticly sample architectures from the SuperNet. To increase the correctness of the evaluation of architectures, besides direct evaluation using the inherited weights, we further apply a few-shot predictor to assess the architecture on the other hand. The final evaluation of the architecture is the combination of direct evaluation, the prediction from the predictor and the cost of the architecture. We regard the evaluation as a reward and apply a self-critical policy gradient approach to update the architecture probabilities. To further reduce the side effects of weight sharing, we propose a weakly weight sharing method by introducing another HyperNet. We conduct experiments on datasets including CIFAR-10, CIFAR-100 and ImageNet under NATS-Bench, DARTS and MobileNet search space. The proposed WPNAS method achieves state-of-the-art performance on these datasets.


Introduction
Neural Architecture Search (NAS) is proposed to make the computer automatically search for an optimal neural network architecture, whose performance can surpass the manually designed state-of-the-art neural network architectures and has similar or less computational complexity (e.g, latency, FLOPs) [51]. Early NAS methods usually sample architectures from a pre-defined search space, obtain the performance of the architecture after training from scratch, and update the search algorithm through reinforcement learning [34,51] or evolution algorithm [27,46] until one or a cluster of optimal architectures are obtained. Although these methods have achieved amazing performances, they often need thousands of GPU-day to run the searching algorithm. Weight-sharing [23,26] and predictor [25,38] are two most commonly used methods to solve the above problem of excessive resource consumption.
Weight sharing refers to building a SuperNet and all subarchitectures in the search space can inherit weights from the SuperNet. Therefore, the amount of parameters to be trained in the search process and the search time can be greatly reduced. Weight sharing NAS algorithms can be divided into two categories: i) Gradient based which parametrize architecture parameter as part of the SuperNet and update the parameter of the SuperNet (architecture parameters and weight parameters) in a differentiable way [8,23,42,43], ii) Sampling based algorithms including uniform sampling [7,14] and learning based sampling [1,5,44]. Because all sub-architectures are deeply coupled with each other, the evaluation of the performance of each architecture have large gaps between those architectures obtained by stand-alone training [7]. Therefore, in order to improve the ranking consistency between the architectures of weight sharing and the architectures obtained by stand-alone training, there are still two major problems to be solved: i) how to improve the correctness of the evaluation of each architecture and ii) is there any way to reduce the degree of weight sharing.
Predictor based NAS algorithms [25,38] are proposed thanks to the introduction of several excellent benchmarking datasets such as NAS-Bench-101 [48], NAS-Bench-201 [11] and NATS-Bench [9]. Predictor based NAS collects several architecture-accuracy pairs as training data in advance, and uses MLP [38] or GCN [25] as predictor to train the direct mapping from architecture to accuracy. However, collecting architecture-accuracy pairs is a resource consuming issue, because it needs to fully train each architecture from scratch of the training pairs. How to improve the training efficiency of the predictor to reduce the dependence on training data is also an important question to be answered.
In this paper, we attempt to answer the above questions using a Weight-sharing and Predictor based NAS (WPNAS) method. First, we construct a SuperNet in a weight-sharing way and probabilistic sampling architectures from the Super-Net with inherited weights. To increase the correctness of  Figure 1. Overview of the proposed WPNAS. We apply a probabilistic sampling strategy to sample architectures from the SuperNet and use a HyperNet to alleviate the influence of weight sharing. The evaluation of the sampled architecture comes from three parts: direct evaluation, prediction from the few-shot predictor and the FLOPs of the architecture. Then we regard the evaluation of the architecture as reward and use a policy gradient algorithm to update architecture probabilities. The combined reward contains information from both SuperNet and predictor, thus may lead better evaluation. Few-shot learning is proposed to reduce the demand for training data of the predictor.
the evaluation of architectures, besides direct evaluation, we further apply a predictor to predict the performance the architecture on the other hand. Direct evaluation from SuperNet, the prediction from the predictor and the cost of the architecture are regarded as the reward and we apply a self-critical policy gradient approach to update the architecture probabilities. To increase the training efficiency of the predictor, we propose a few-shot learning based strategy to train the predictor. Besides, to further reduce the negative effects of weight sharing, we propose a weakly weight sharing method. Namely, we feed the sampled architecture into a HyperNet and generate different offset-weights, the final weights is the product of the offset-weights and weights from SuperNet. By this way, the degree of weight sharing is reduced. In summary, the contribution of this work contains four major parts: • We jointly use weight-sharing and predictor and use a self-critical policy gradient algorithm with probabilistic sampling to update architecture parameters.
• We propose a few-shot learning based predictor to increase the training efficiency of the predictor.
• HyperNet based weakly weight sharing strategy is proposed to alleviate the influence of weight sharing.
• We conduct comprehensive experiments on CIFAR and ImageNet datasets under several search space and the results demonstrate the superiority of the proposed method.

Related Work
Weight sharing based NAS. ENAS [26] is the first NAS paper to apply weight sharing to address the high searching cost problem. ENAS [26] applies reinforcement learning to train a controller to sample architectures and it does not need to train each sampled architecture from scratch, but directly inherit parameters from the Supernet, so the architecture search time can be reduced from thousands of GPU-day to only less than a single GPU-day. DARTS [23] uses a continuous relaxation on the categorical choices of inputs and operations. The weight parameters and architecture parameters can be updated based on the gradient, and the search time can be further reduced. DARTS has received a lot of attention and follow-up research because of its high simplicity and effectiveness [3,6,8,42,43]. The above gradient based methods still face two important problems: high GPU memory occupation and lack of theoretical support for discretization. [10] and [2] propose to apply Gumbel-Softmax trick to reduce the GPU memory occupation. Sampling based methods are proposed to solve the above two problems at the same time. SPOS [14] proposes a two-stage algorithm that abandoning the architecture parameters, using only uniform sampling to train the Supernet, and applying evolution algorithm to search optimal architectures on the  trained Supernet which can also achieve excellent performance. Some follow-up studies also improved this unbiased sampling method from many aspects [7,20,49]. Learning based sampling can favor the most promising architectures and learn the optimal sampling probabilities. PARSEC [1] and FP-NAS [44] apply probabilistic sampling and use an empirical Bayes Monte Carlo procedure to optimize architecture parameters. MCT-NAS [31] challenges the operation independency assumption and applies a Monte Carlo tree search strategy which captures the dependency among layers and operations. Our proposed method falls into the probabilistic sampling based NAS method category, but we use a different self-critical policy gradient way to optimize the architecture probabilities. Predictor based NAS. Predictor based NAS methods are proposed recently thanks to the introduction of some benchmark dataset. [38] first apply a predictor to regress the validation accuracy. They train several architectures to obtain architecture-accuracy pairs as training data to train the regression model to predict the validation accuracy directly. The architecture with the best predicted accuracy is trained from scratch for the final deployment. [12,25] applies Graph convolutional networks (GCN) to embed the architectures to better capture the topology information. [4,41] considers the ranking property between architectures and apply rank loss to optimize the predictor. [35] using semi-supervised learning to reduce the dependency on training samples. In this paper, we reduce the demand for training samples from another perspective by using few-shot learning.

WPNAS
In this section, we will present our WPNAS algorithm. In Section 3.1, we introduce the probabilistic sampling method; in Section 3.2, we describe the novel few-shot predictor; in section 3.3, we introduce the weakly weight sharing approach; in Section 3.4, we summary the optimization algorithm.

Probabilistic Sampling
Suppose the SuperNet with model weights ω with a macro search space has L layers, each layer have K candidate operations [1,44], we also introduce architecture parameters (i.e. architecture probabilities) α and a prior distribution p(A|α). Each architecture A is sampled based on p(A|α), thus the task of architecture search is to learn the distribution p(A|α) which prefers better architectures. Suppose the probabilities at different layers are independent to each other, the probability of architecture A being sampled can be calculated as below:

Few-shot Predictor
Traditional accuracy predictor of NAS always needs thousands of training data pairs which require tremendous computing resources to obtain. To address this problem, we borrow the idea from few-shot learning on image classification for more efficient predictor training. Specifically, we construct a simple yet effective Relation Network [32]-like predictor by learning to compare query architectures and support architectures. The predictor has two major modules: architecture encoding module and architecture decoder module (see Figure 2.). The architecture encoding module embeds the support architectures and query architecture into the hidden representations with the same dimension. The embeddings from support set and query architecture are concatenated and are fed to the architecture decoder module to learn the accuracy offset between each architecture in the support set and the query architecture. The encoding and decoding modules can be MLP based, RNN based, Transformer based or GCN based. During training, the support set and the query set are randomly chosen from the training data. Once trained, the few-shot predictor is able to predict N3 N2  new architectures by computing relation offsets between the query architecture and the few examples of training set. For simplicity, we denote the output of the few-shot predictor as F SP (A). In the following section, we will demonstrate the superiority of the proposed few-shot predictor than supervised learning based predictor.

Weakly weight sharing
Although weight sharing brings benefits such as reducing search consumption, it also leads to the fatal problem of the ranking gap between architectures trained by weight sharing and stand-alone training. To relieve this problem, we apply a weakly weight sharing strategy by introducing a HyperNet (see Figure 3.). Specifically, for a sampled architecture A = (A 1 , ..., A L ), the weights of the architecture can be inherit from the SuperNet, and we call these weights as base weights A bidirectional LSTM (bi-LSTM) with L time steps is used as the HyperNet to generate several weights for each edge (or layer) given the architecture, and these weights are called offset weights . The final weights for evaluation and back-propagation are the product of the based weights and offset weights: Suppose two architectures choose the same operation in a certain layer. If there is no weakly weight sharing, the two architectures have exactly the same weights on that layer. After the introduction of HyperNet, different architectures will generate different offset weights, if the two architectures have the same operation in a certain layer, their weights at that layer will be different. Thus, the introduction of HyperNet will reduce the degree of weight sharing among architectures, which may alleviate the problem of the ranking gap. The weights of HyperNet can be fixed or updated simultaneously with the SuperNet.
The maximum kernel size we used in this study is 5, so the dimension of ω l o is 5 × 5 = 25. For operations with smaller kernel size such as 3 × 3, the weights of the top left corner are cropped and used.

Optimization
Given a classification task with input X and targets y, the goal of optimization can be present as: We can simplify the optimization through Monte Carlo sampling and policy gradient algorithm. For ω, the optimization is straight forward. For a sampled architecture A with weights ω, we can directly optimize ω by ∇ ω log p(y|X, A, ω).
For α, the gradient of loss can be estimated by policy gradient: where r is the reward function. The reward function contains three parts: 1) direct evaluation, i.e. the negative crossentrophy loss of the architecture log p(y|X, A, ω) ; 2) the output of the few-shot predictor F SP ( A) ; 3) and the negative FLOPs of the sampled architecture −F LOP s( A, X). Thus, we get: where β 1 and β 2 are two hyper-parameters to control the weights of each part. Because the reward function of policy gradient is scalar, the information from predictor and FLOPs can be easily included when updating the architecture probabilities. Inspired by [28], we also add a baseline item in the reward function from the architecture obtained by greedy decoding. Specifically, we apply an arg max operation on the probability distribution:Â = arg max A p(A|α) to get the architectureÂ and weightsω. The baseline item is: Thus, the final optimization equation of self-critical training is: At initialization, the probability distribution is set to uniform distribution for fair search. During training, the probability distribution α is optimized to prefer architectures with higher accuracy from both SuperNet and the predictor, and lower FLOPs. Once trained, the final architecture is discretized by A f inal = arg max A p(A|α). Unlike SPOS [14], our method does not require a second stage to find the optimal architecture.

Experiments
To validate the advantages of the proposed algorithm, we conduct experiments on three commonly used search spaces: NATS-Bench search space [9] (section 4.1), DARTS search space [23] (section 4.2) and MobileNet search space [30] (section 4.3). WPNAS procedure. The whole process of our WP-NAS algorithm contains 5 steps. 1) WarmUp. We train the Su-perNet without optimizing the architecture probabilities for 200 epochs. Only weights are updated in this step. We apply SGD optimizer with initial learning rate 0.1 and follow a cosine schedule to decay the learning rate to 0.01. 2) Acquisition of the ground truth of the Predictor. Uniform sampling 200 architectures, inheriting weights from the Su-perNet and train these architectures for 30 epochs. Thus, we can obtain 200 architecture-accuracy data pairs as the ground truth of the predictor. We use the proposed way to get the ground truth instead of training from scratch for the sake of saving time. The Kendall Tau between our GT and that obtained by training from scratch is: Kendall Tau = 0.7, but the training time reduced by 85%. 3) Train the predictor. We use the proposed few-shot learning based algorithm to train the predictor. We split the 200 GTs with 170 architectures for training and 30 architectures for validation. The support size is set to 30. We train the predictor for 100 epochs. 4) WPNAS Search. Alternatively updating the weights and architecture probabilities for 300 epochs. Architecture probabilities are updated based on equation (7) by self-critical policy gradient training with a combined reward of direct evaluation, output of the predictor and the FLOPs. We also apply an SGD optimizer with cosine decay scheduler from 0.01 to 1e-5. 5) Evaluation. Obtain the final searched architecture with an argmax operation and train this architecture from scratch for 600 epochs.

NATS-Bench experiments
NATS-Bench. NATS-Bench [9] is an extension version of NAS-Bench-201 [11] and it is a unified benchmark on searching for both topology and size. For topology search space (TSS), it is a cell-based search space inspired by some cell-based algorithms [23,26]. A cell is represented as a densely-connected directed acyclic graph (DAG) and all cells have the same topology. The DAG contains 4 nodes and 6 edges, and each edge of the DAG has 5 candidate operations. Thus, there are 5 6 = 15625 different architectures in the TSS search space. For size search space (SSS), the cell is the best one in the TSS search space on CIFAR-100 dataset. Each architecture contains five layers with a unique configuration based on the number of channels in each layer. There are 8 candidate channel numbers for searching: {8, 16, 24, 32, 40, 48, 56, 64}. Therefore, the search space size are 8 5 = 32768. We perform experiments on CIFAR-10 and CIFAR-100 datasets [21]. Results on NATS-Bench. Table 1. shows the results on NATS-Bench. Baselines. We apply single-path one-shot (SPOS) [14], the most representative sampling based algo-   rithm as the baseline. We train the SuperNet for 500 epochs with uniform sampling. Similar as [14], we use an evolutional algorithm to find the optimal architecture. Another baseline is adding landmark regularization [50] when training the SuperNet. Since we have several GTs of the predictor, we can make use of these data as the landmark to better guide the ranking between architectures during SuperNet training. We also train the SuperNet for 500 epochs with landmark regularization and also apply a second stage to find the optimal architecture. Metrics. We report the ranking correlation in terms of two metrics: Kendall-Tau and mean square error (MSE). We also report the rank and accuracy of the searched architecture and the overall cost. We gradually add probability sampling, few-shot predictor and weakly weight sharing mechanism, and compare the results with baselines. Experiments on CIFAR-10 and CIFAR-100 under TSS and SSS search space have consistent results, that is, with the introduction of the proposed novel methods, the ranking performance is gradually improved, and the ranking and accuracy of the final architecture are also improved accordingly. It should be noted that after the introduction of each method, the cost will increase accordingly, especially after the introduction of predictor. However, compared with baseline, the cost is also in the same order of magnitude and less than 1 GPU-day, which is also acceptable. Few-shot Predictor. Table 2. shows the results of the proposed few-shot predictor. The baseline is supervised training whose input is a single architecture and the output is a scalar (validation accuracy). We test three types of the predictor, which are MLP, RNN and Transformer based. For both supervised learning and few-shot learning, the training / validation set are both 170/30. We train the predictor for 100 epochs using SGD with learning rate 1e-4. We apply a ranking loss similar as [41]. We report three metrics: Kendall-Tau, Pearson Correlation Coefficient and Mean Square Error (MSE).
The results show that few-shot predictor has consistent better Kendall-Tau and MSE on CIFAR-10 and CIFAR-100 under TSS and SSS search spaces. The MSEs of few-shot predictor are extremely lower than that of supervised learning, because the proposed few-shot predictor can better learn the relationships between architectures, even only ranking loss is used without any other regression guidance.

DARTS search space
To compare with other NAS algorithms, we also conduct experiments on the commonly used DARTS search space [23] on CIFAR-10 and ImageNet [29] datasets. DARTS search space is also a cell based search space. The DAG contains 4 inner nodes, and each node has two parents nodes. The search space size is 3.3 × 10 13 . We set two types of architecture parameters for each node of the DAG: operation selection and parent node selection. Operation selection is used to select one operation from 8 candidates. Parent node selection is designed to select two parents nodes from the current node's all previous nodes. Suppose the current node has N previous nodes, the number of candidate combinations are C 2 N . Suppose operation selection and parent node selection are independent to each other, equation (1) remains valid as well. All the rest experiment settings are identical to DARTS for fair comparison. We compare our method with several manual networks such as DenseNet [19] and MobileNet [17] and some SOTA NAS algorithms on DARTS search space [3,8,23]. Similar as DARTS, we conduct the search experiment on CIFAR-10 dataset and validate the performance of the searched architecture on CIFAR-10 and ImageNet dataset. The channel number is expanded on Ima-geNet. Similar as [8,23], we also run 3 times evaluation on CIFAR-10 to get a more accurate result. Table 3. shows the results on DARTS search space. WPNAS-A and WPNAS-B are two searched architectures with different β 2 (WPNAS-A: β 2 = 1 −3 , WPNAS-B: Similar with [7,8], we also add some tricks such as Swish, SE and AutoAugment, the performances of WPNAS-A and WPNAS-B are all improved significantly and achieves SOTA performance.

MobileNet search space
MobileNet search space is another commonly used macro search space for NAS benchmarking. Following [31], we construct the MobileNet-like SuperNet with 17 search blocks, and each block is a MobileNetV2 inverted bottleneck [30]. For each search block, the candidate set of convolutional kernel size is {3,5}, the expansion ratio candidate set is {1,3,6}, nonlinearities is selected in {ReLU, HardSwish [16]} and each block can choose to use Squeezeand-Excitation (SE) [18] module or not. The search space size is 24 17 ≈ 2.9 × 10 23 . We compare our NAS algorithm with some SOTA algorithms on this search space including SPOS [14], FBNet [36,39], FairNas [7], FP-NAS [44] and MCT-NAS [31]. We also conduct the search experiment on CIFAR-10 dataset and transfer the architecture on ImageNet. The channel number is expanded on ImageNet. Table 4. shows the results on MobileNet search space. We also obtain two final architectures WPNAS-A and WPNAS-B (WPNAS-A: β 2 = 1 −3 , WPNAS-B: β 2 = 1 −5 ). On ImageNet dataset, WPNAS-A has higher top-1 accuracy (75.6) compared with other methods (SPOS, FBNet and Fair-NAS) with similar Params and FLOPs. WPNAS-A is a bit lower than MCT-NAS-C, but MCT-NAS use other training tricks including WarmUp and EMA [15]. WPNAS-B also has higher accuracy than FP-NAS with similar Params. FB-NetV2 has higher accuracy, but its cost (0.6k) is also much higher than ous. With the help of AutoAugment, WPNAS-B (77.8) also surpass FairNAS † (77.5).    Table 4. Search results on CIFAR-10 and ImageNet under MobileNet search space and comparison with state-of-the-art methods. † represents using Swish, SE and Autoaugment. represents using training tricks including WarmUp and EMA [15].

Conclusion
In this paper, we propose to jointly use weight sharing and predictor in a unified framework. Architecture coupling caused by weight sharing can lead to big evaluation ranking gap between architectures using weight sharing training and stand-alone training. To increase the correctness of the evaluation of architectures, besides the direct evaluation using the weights from SuperNet, we also introduce a predictor to evaluate the architecture from another perspective. The evaluations from these two parts are combined as the final prediction of the architecture, and are used as the reward of a self-critical policy gradient algorithm. After the introduction of predictor, the evaluation of architecture will be more accurate, which can also better guide the NAS algorithm to find better architectures. Besides, we propose a novel fewshot learning based predictor to enhance the performance of the predictor. Furthermore, we propose a weakly weight sharing strategy by introducing a HyperNet to reduce the degree of weight sharing, thus leading to more stand-alonetraining like SuperNet training. We conduct comprehensive experiments on CIFAR and ImageNet datasets under NATS-Bench, DARTS and MobileNet search space and our algorithm achieves SOTA performances on these datasets.