{'fraction_non_alphanumeric': 0.03692471401968608, 'fraction_numerical': 0.02239957435488162, 'mean_word_length': 4.763876111622202, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
How Does Supernet Help in Neural Architecture Search?

Weight sharing, as an approach to speed up architecture performance estimation has received wide attention. Instead of training each architecture separately, weight sharing builds a supernet that assembles all the architectures as its submodels. However, there has been debate over whether the NAS process actually benefits from weight sharing, due to the gap between supernet optimization and the objective of NAS. To further understand the effect of weight sharing on NAS, we conduct a comprehensive analysis on five search spaces, including NAS-Bench-101, NAS-Bench-201, DARTS-CIFAR10, DARTS-PTB, and ProxylessNAS. We find that weight sharing works well on some search spaces but fails on others. Taking a step forward, we further identified biases accounting for such phenomenon and the capacity of weight sharing. Our work is expected to inspire future NAS researchers to better leverage the power of weight sharing.


INTRODUCTION
Over recent years, weight-sharing NAS has emerged as a way to search for architectures under very limited resources, and has been gaining popularity (Pham et al., 2018;Cai et al., 2019;Bender et al., 2018). It builds all architectures into a supernet where each architecture is a submodel. Once the supernet has been trained, architectures can be directly evaluated by inheriting weights from the supernet, thus saving the huge cost to train each architecture from scratch.
Weight sharing is built on the hypothesis that an architecture's performance estimated within a supernet indicates its real performance. However, such hypothesis has been long questioned. Many analytical papers have conducted empirical analysis on whether and how weight sharing helps in finding the best architecture (Bender et al., 2018;Luo et al., 2019;Yu et al., 2019;Yang et al., 2019a;Zhang et al., 2020;Pourchot et al., 2020;Yu et al., 2020;Bender et al., 2020). Though valuable insights have been delivered, the understanding of weight sharing still only touches a tip of the iceberg. Futhermore, due to the differences in search spaces and experiment setups, sometimes different works even draw contradictory conclusions (Bender et al., 2018;Li & Talwalkar, 2019).
For a deeper understanding, we conduct a more comprehensive study spanning five search spaces: NAS-Bench-101 (Ying et al., 2019), NAS-Bench-201 (Dong & Yang, 2020), DARTS-CIFAR10, DARTS-PTB , and ProxylessNAS (Cai et al., 2019). The study reveals that, for some spaces, NAS can benefit from weight sharing to find a good-performing architecture, while for others, it hardly works. We then analyzed the results to dig out the reason for such phenomenon. Our experiments exclude the factor of sharing density of weights and training instability, and point the cause directly to the bias towards certain operators, connections and model sizes. We noticed that weight sharing performs better when the bias coincides with the actual rank of architectures.
Since weight sharing struggles to identify the best architectures, some works resort to pruning search space in order to increase the probability of finding the best, e.g., RL-based methods (Pham et al., 2018;Bender et al., 2020) and heuristic methods (You et al., 2020;Hu et al., 2020;Li et al., 2019). To identify the upper limit to which weight sharing can correctly prune the search space, we compared weight sharing on pruned NAS-Bench series. We came to a conclusion that weight sharing is able to find top-10% architectures, but struggles to find the best.
To conclude, our contributions can be summarized as follows: 1 arXiv:2010.08219v2 [cs.LG] 5 May 2021 2nd Workshop on Neural Architecture Search at ICLR 2021 • We conduct a comprehensive study on five different search spaces, spanning from CNN to RNN. The pairwise comparison between different spaces reveals more convincing insights, and also explains some contradictory conclusions drawn from previous papers. • To the best of our knowledge, we are the first to systematically analyze the performance of weight sharing on pruned search space. • We released a NAS benchmark containing more than 500 architectures where the ground truth 1 of each architecture is also reported, which we expect will facilitate NAS research. 2

WEIGHT SHARING BENCHMARK
In this section, we conduct a comprehensive study of weight sharing on five search spaces. For all the experiments, we adopt uniform sampling to train a supernet, i.e., single-path one shot (SPOS) (Guo et al., 2019), as it is a generalized approach and it can also be seen as a basic component of other advanced sampling methods that exclude other training tricks. Details about experiment setups including search spaces and evaluation methodology can be found in Appendix A.
Longer training does not guarantee better results. As previous works (Chu et al., 2019b;a;Guo et al., 2019;Luo et al., 2019) pointed out that supernet needs to be trained sufficiently to benefit NAS, we first vary the training epochs exponentially instead of using a fixed training schedule. From the perspective of average validation performance, supernet converges well and there is no sign of over-fitting (see Appendix D). However, as shown in Figure 1, a longer training does not guarantee a better ranking model. For example, on DARTS-CIFAR10, correlation achieves 0.54 when trained with 64 epochs, but later decreases to around 0.2 with longer training. On PTB, the correlation can be even worse than 0. There is a similar phenomenon on top-k. Although top-k is not very sensitive to the changes in correlation, performance of selected architecture could still drop when supernet is trained longer.  Figure 1: We show correlation, top-k (i.e., best ground truth of k architectures selected with the help of weight sharing) and random top-k (i.e., best ground truth of randomly-selected k architectures). We also show the best ground truth (in green dash-dot line) if the metric is available. Details explanation on metrics are in Appendix A.
Effectiveness of weight sharing relies on search space. We draw the conclusion by comparing the results across different search spaces. For example, on NAS-Bench-201, the correlation can be as high as 0.7, whereas using weight-sharing to guide NAS on DARTS-PTB can be even worse than  Figure 2: Comparison between random search and weightsharing NAS as we decrease the sharing density by conducting NAS on a randomly-downscaled search space. We show top-10 accuracy (blue and red lines) for weight sharing and random search, respectively, and correlation in black lines, when search space is k times smaller than the original one. The error-bars and regions are min/max across 3 different runs.
Reducing the sharing density hardly helps. Zhang et al. (2020) pointed out the intuition that submodels have interference with each other, and thus proposed to reduce the density of weight sharing (i.e., how many different architectures are sharing the weight of one operator). However, our experiments show that this is only true when the sharing density is substantially low. For example, in Figure 2, the improvement is only noticeable when the downscaling factor is up to 64 on NAS-Bench-201, at which time the search space only contains 244 different architectures. On NAS-Bench-101, no increasing trend is observed at all.
Weight sharing imposes a consistent bias. One may attribute the failure of weight sharing to the instability of random-path gradient descent, bad hyper-parameters. But according to our study, that is not the case. As shown in Figure 3, despite trained with different epochs, supernets are highlyagreeable when ranking architectures, even though they can be unanimously wrong (e.g., negative GT-correlation on DARTS-PTB). Similar consistency are also seen between different datasets, hyper-parameters and training tricks (Appendix C). The bias imposed by weight sharing seems consistent and lies in the intrinsics of weight sharing itself.  : Relationship between supernet performance and ground truth performance, grouped by decision on a specific operator, as stated in parenthesis. The most biased candidate is highlighted.

Weight sharing favors big models.
Architectures with large number of parameters (or FLOPs) usually have better ground truth accuracy . As a result, for a search space spanning both very small and very large models (e.g., 1M to 13M parameters for NAS-Bench-101, 0.1M to 1.3M for NAS-Bench-201), model size can be a fairly good performance estimator baseline. As shown in Figure 3, the correlation between ground truth performance and parameter size on NAS-Bench-101 can be as high as 0.67, outperforming all the attempts from weight sharing to estimate a good rank. Moreover, We find that weight sharing performs better in search spaces with higher GT-Param correlations. Sometimes, weight sharing even has a higher correlation with model size than its correlation with ground truth, in which case, weight sharing learns to find big models, rather than good-performing models.
Weight sharing is biased towards certain operators. We support this claim by reviewing the architecture distribution grouped by decisions of operators. Some typical cases are shown in Figure 4. For NAS-Bench-101, Conv3x3 is clearly preferred by weight sharing. Fortunately, architectures with Conv3x3 indeed have relatively good performance and that is why top-k on NAS-Bench-101 is fairly good. On NAS-Bench-201, weight sharing still favors Conv3x3 as the operator connecting input node to output node of a cell, but almost all architectures with top ground truth accuracy choose the skip connection instead. This accounts for the fact that top-10 is almost no better than top-1 in Figure 1. Similarly, on DARTS-PTB, Tanh, the activation function which is favored by weight sharing, turns out to be a worse option. : Comparison between random search and weight-sharing NAS on best-architecture subsets of search spaces. When search space is k times smaller than the original one, we show top-10 accuracy (blue and red lines) for weight sharing and random search, respectively, and correlation in black lines. The error-bars are the min and max across 3 different runs.

CAPACITY OF WEIGHT SHARING
Since identifying the top architectures with weight sharing is difficult, previous works resort to weight sharing as a method to prune the worst, hoping to improve the performance distribution of search space, thus increase likelihood of finding the best. On search space weight sharing still works reasonably well (NAS-Bench-101 and 201), we ask the following question: is weight sharing still effective on a pruned search space? To answer the question, we test the capacity of weight sharing, after which prune factor NAS with weight sharing does not gain more benefits.
In particular, we run experiments on sub-search-space of NAS-Bench-101 and 201, by keeping a subset of architectures with the best ground truth performance (detailed setups in Appendix A). We vary the subset size and plot the figures shown in Figure 5. On NAS-Bench-101, we can see the advantage of weight sharing against random search becomes less obvious as the correlation goes down. This aligns with the intuition because the best architectures have very close accuracy and are hard to distinguish. Interestingly, on NAS-Bench-201, the advantage of weight sharing against random search is neither steadily increasing, nor gradually diminishes. It seems that weight sharing is reaching its limit when the prune factor is 8x, at which point, the advantage of weight sharing compared to random search becomes very little. However, after this point there could still be opportunities for weight sharing, as the correlation grows up to 0.6 when the space is pruned 128 times smaller. We conjecture that this phenomenon is a result of certain operators biased by weight sharing being removed from search space (e.g., the bad-performing Conv3x3 on NAS-Bench-201).

CONCLUSION
In this paper, we give a comprehensive view of weight sharing in neural architecture search. Through our experiments, we have shown that weight sharing, though effective on some search spaces, can be limited on others. Weight sharing imposes a bias towards certain architectures and its capacity is limited to discriminate better architectures from worse. Our findings call for a better understanding of such biases and co-design of search space and weight sharing NAS. Meanwhile, instead of directly identifying the top, weight-sharing-guided search space pruning is shown to be a promising lead, and we hope this will inspire future work. NAS-Bench-201. NAS-Bench-201 (Dong & Yang, 2020) constructs its cell with 4 nodes connected with 6 edges, with an operator choosing from 5 candidate operators on each edge. In NAS-Bench-201, 341 out of 15625 are invalid, where input node of the cell is completely not connected to output node, we simply devise a filter to avoid those invalid cells from being chosen. Though the original NAS-Bench-201 is given across three different scaled image classification datasets, we use CIFAR-100 only as architecture rank on three datasets are reported to be highly correlated and the other two datasets have already been used on other search spaces.
DARTS-CIFAR10. The search space on CIFAR-10 dataset adopted in DARTS  is very similar to the ones used by Pham et al. (2018); Zoph et al. (2018), except that all nodes in a cell are concatenated into the output and there are subtle differences in convolution with stride = 2 in reduction cell. Previous works handle the search space differently. For example, DARTS expands the combinations of operators and inputs, thus the supernet is 3 -4 times larger than that constructed following ENAS (Pham et al., 2018). In our paper, we use the straight-forward supernet-building which is proposed in ENAS.
ProxylessNAS. We follow ProxylessNAS-GPU (Cai et al., 2019) setting, with 1.35 width multiplier. During training, all architectures are equally likely to be sampled. In evaluation, we only consider architectures that fit in mobile settings and filter out the architectures with FLOPs greater than 600M.

DARTS-PTB.
We use the RNN cell search space consisting of 8 choices of activation function and choices of predecessors, following . We use the official train, valid and test split provided by Penn Treebank.
We present a summary of all search spaces in Table 1.

A.2 EVALUATION METHODOLOGY
Single-path one shot. In single-path one shot, an architecture is sampled per mini-batch with the objective of minimizing the expectation of loss, i.e., where A is the architecture search space, φ is the supernet weights, and N is the corresponding architecture using weights from supernet. When evaluating a submodel, we first extract it from supernet inheriting supernet weights, calibrate Batch-Norm statistics (Guo et al., 2019;Bender et al., 2018;Pourchot et al., 2020;Yu et al., 2020), and then evaluate it on the full validation set (top-1 accuracy for image classification task and perplexity for language modelling). When calculating the metrics (including correlation and top-k), we sample and evaluate 1k architectures for NAS-Bench series, 100 for DARTS series, and 20 for ProxylessNAS.
Hyper-parameter settings. We list important hyper-parameters for supernet training in Table 2. We run the training with 1000 epochs if not otherwise specified. For ground truth training, we completely followed the settings mentioned in the original paper, except those which are tuned to fit into our GPU capacity (batch size = 96 for DARTS-CIFAR10, and 1024 for ProxylessNAS).
Metrics. Following previous works, we use the following metrics to assess the effectiveness of weight sharing in NAS: •  (2019), we show the correlation between supernet performance and ground truth performance. We will use Spearman correlation (Spearman, 1961) in this paper. A higher correlation means better consistency. Correlations are calculated on a randomly sampled subset (1k for NAS-Bench series, 100 for DARTS series, 20 for ProxylessNAS).
• Top-k: We randomly sample 1k architectures, choose top-k based on their performance evaluated on supernet, and obtain their ground truth performance. We then use report test performance of the architecture with the best ground truth performance on validation set.
Search space pruning. To keep the best performing subset of search space, we adopt the method to keep the architectures with best ground truths. In practice, the ground truths are usually not available and replaced with the estimation by weight sharing methods (Pham et al., 2018;You et al., 2020). Note that we did not prune the search space at operator level, as previously done in Chen et al. (2019); Hu et al. (2020).

B PERFORMANCE BENCHMARK
For the purposes of conducting multiple experiments mentioned in our paper, we trained 377 architectures on DARTS-CIFAR10, 36 architectures on ProxylessNAS and 137 architectures on DARTS-PTB, independently from scratch. We show the distribution in Figure 6, where each point corresponds to an architecture, and publish these results for the usage of future researches.

C EFFECT OF SUPERNET TRAINING TRICKS
Most of the experiments in this paper are run with the training approach and hyper-parameters listed in Appendix A. However, recent works have proposed many variations based on SPOS (Yang et al., 2019b;Pham et al., 2018;You et al., 2020;Chu et al., 2019b). Even if we faithfully follow SPOS to train a supernet, there are still quite a few hyper-parameters to tune. We test the impact of different settings and show the results in Table 3 and Figure 7. Judging from the results, there is no evidence that applying any of the tricks is beneficial. Meanwhile, almost all weight sharing methods are correlated with each other, implying that the bias of weight sharing is consistent. Table 3: Comparison of correlation with ground truth (Corr-GT), top-10, and correlation with baseline (Corr-BL) after varying training settings and applying training tricks.

D CONVERGENCE OF SUPERNET
Here, we discuss the convergence of supernet. We computed the average supernet performance, i.e., average accuracy of submodels evaluated on supernet. As shown in Figure 8, there is a clear trend that extended training will boost supernet performance on all five search spaces and the performance is expected to continue to increase the training epochs. There is no sign of over-fitting. On NAS-Bench-101, supernet performance even surpasses average ground truth. Better average accuracy implies that the validation loss in Equation 1 is optimized better. However, this does not mean the supernet is more capable of identifying top architectures, as the correlation sometimes goes lower.