{'fraction_non_alphanumeric': 0.03281987769951235, 'fraction_numerical': 0.022370152488582706, 'mean_word_length': 4.610596409959467, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EcoNAS: Finding Proxies for Economical Neural Architecture Search

Neural Architecture Search (NAS) achieves significant progress in many computer vision tasks. While many methods are proposed to improve the efficiency of NAS, the search progress is still laborious because training and evaluating plausible architectures over large search space is time-consuming. Assessing network candidates under a proxy (i.e., computationally reduced setting) thus becomes inevitable. In this paper, we observe that most existing proxies exhibit different behaviors in maintaining the rank consistency among network candidates. In particular, some proxies can be more reliable – the rank of candidates does not differ much comparing their reduced setting performance and final performance. In this paper, we systematically investigate some widely adopted reduction factors and report our observations. Inspired by these observations, we present a reliable proxy and further formulate a hierarchical proxy strategy that spends more computations on candidate networks that are potentially more accurate, while discards unpromising ones in early stage with a fast proxy. This leads to an economical evolutionary-based NAS (EcoNAS), which achieves an impressive 400×search time reduction in comparison to the evolutionary-based state of the art [19] (8 v.s. 3150 GPU days). Some new proxies led by our observations can also be applied to accelerate other NAS methods while still able to discover good candidate networks with performance matching those found by previous proxy strategies. Codes and models will be released to facilitate future research.


Introduction
Neural Architecture Search (NAS) has received wide attention and achieved significant progress in many computer vision tasks, such as image classification [11,23,30,35], detection [6,9,28], and semantic segmentation [17]. Although recent NAS methods [18,20,22] improve the search * Equal contribution.   [15]. Simply replacing the original proxy with a more consistent and efficient one could reduce the search cost of DARTS [20] and AmoebaNet [23]. The proposed EcoNAS uses an efficient proxy with a novel hierarchical proxy strategy, reducing around 400× search cost and achieving competitive performance comparing with AmoebaNet [23]. efficiency from earlier works [34], the search progress is still time-consuming and requires vast computation overhead when searching in a large search space since all network candidates need to be trained and evaluated.
A widely adopted approach to alleviating this problem is by training and evaluating network candidates under proxies (i.e., computationally reduced settings [20,22,23,35]). There are four common reduction factors, namely, the number of channels for CNNs (c), the resolution of input images (r), the number of training epochs (e), and the sample ratio (s) of the full training set. These reduction factors either reduce the computation costs of networks or reduce the training iterations to save the search time. While introducing proxies sounds appealing, a caveat is that different reduction factors exhibit different behaviors in keeping the rank consistency among network candidates. We observe that some architectures applied with certain reduction factors win in the reduced setting but perform worse in the original setup on CIFAR-10 [15]. Some other reduction factors perform more consistently. Those consistent and efficient reduced settings accelerate existing NAS methods and achieve competitive results as shown in Fig. 1.
In this work, we investigate the behaviors of reduction factors [c, r, s, e]. Two main observations were obtained: (1) with the same iteration numbers, using more training samples with fewer training epochs could be more effective than using more training epochs and fewer training samples in reducing the rank inconsistency; (2) reducing the resolution of input images is sometimes feasible while reducing the channels of networks is more reliable than reducing the resolution. The aforementioned observations motivate us to design reliable proxies that reduce channels and input resolutions while using all training samples. These new proxies apply well to many NAS methods, including evolutionarybased and gradient-based NAS methods [5,20,23]. In particular, we observe consistent search time reduction when these new proxies are applied, while the discovered networks are similarly competitive to those found by the original NAS methods.
We further formulate a novel hierarchical proxy strategy for evolutionary-based NAS. The goal of the hierarchical proxy strategy is to discard less promising candidates earlier with a faster proxy and evaluate more promising candidates with a more expensive proxy. The strategy is both effective and efficient: (1) this design saves substantial computation overhead by saving evaluation on less promising networks, and (2) assigning more resources to more promising networks help us to find good architectures more precisely. Thanks to the hierarchical proxy strategy, the proposed economical evolutionary-based NAS (EcoNAS) enjoys nearly 400× reduced search time (8 vs. 3150 GPU days) in comparison to the evolutionary-based state of the art [23] while maintaining comparable performance.
To summarize, our main contributions are as follows: 1. We conduct extensive experiments to study commonly applied reduction factors. While inconsistency of reduction factors is well-known in the community, our work presents the first attempt to analyze the behaviour systematically. 2. Observations from the experiments lead to some fast and reliable proxies that are applicable to a wide range of NAS methods, including evolutionary-based NAS [23] and gradient-based NAS methods [5,20]. 3. We present a hierarchical proxy strategy that leads to EcoNAS, reducing the search time requirement from thousands of GPU days to fewer than 10 GPU days.

Related Work
Neural Architecture Search. NAS aims at searching for good neural networks automatically in an elaborately designed search space. One common approach of NAS is to train and evaluate each network candidates on a proxy, i.e., computation reduced setting, and to search architectures using either evolutionary algorithms (EA) [2,19,23,24], reinforcement learning (RL) [18,22,33,34,35], or gradientbased methods [20,30]. One-Shot methods [3,4,11] usually train a supernet covering the search space once and then apply search algorithms to search the best path in this supernet as the searched architecture. After the seminal work by Zoph and Le [34] that requires up to hundreds of GPU days to find a good architecture, many NAS methods [18,20,22] try to reduce the search cost through different approaches. The community first turns to searching for primary building cells rather than the entire network [33,35]. Based on the cell search, some approaches try to use performance prediction based on learning curves [2] or surrogate models [18] to expedite the evaluation process. Parameter sharing between child models is also common for acceleration,e.g., in DARTS [20], ENAS [22], and One-Shot methods [3,4,11]. Our EcoNAS is an evolutionary-based NAS method [2,19,23,24]. Different from previous works that fix the proxies when searching for network architectures, we design a fast and consistent proxy to reduce the search time, and apply hierarchical proxy strategy to improve the search efficiency. It is noteworthy that another trend of NAS is to search for efficient architectures [5,13,26,29] by adding constraints during search process such as latency [13,26,29] on specific platforms. This differs to our focus on reducing the search cost by more careful and systematic proxy design.
Accelerating neural network training. Parallel computing [1,31,32] could significantly accelerate the training process of deep neural networks. Although these methods save the training time to minutes, the reduction in training time comes with the cost of thousands of GPUs, and their computation overhead remains large. Many studies [7,21,27] evaluate the performance of different networks at reduced settings, e.g., using smaller images in Tiny ImageNet Challenge 1 , using the encodings of network layers [7], reducing the number of training iterations [21], and reducing samples [27]. These studies assume that a specific reduced setting is sufficiently consistent thus do not evaluate the influence of different reduced settings. In contrast, we extensively evaluate influences from different reduced settings.

Exploration Study
In this study, we investigate the behaviors of different reduced settings. Previous works have not studied such behaviours comprehensively and systematically. To facilitate our study, we construct a model zoo containing 50 networks for comparing the rank consistency before and after apply- ing reduced settings. Each network in the model zoo is a stack of cells with alternating normal cells and reduction cells [18,22,35], generated by random sampling. The cell can be regarded as a directed acyclic graph consisting of an ordered sequence of nodes. Each node is a feature map in convolution networks and each directed edge is associated with an operation to be searched or randomly sampled. The entire network structure and one example of the cell are shown in Fig. 2(a) and Fig. 2(c), respectively. The details are provided in the Appendix.

Reduction Factors
In our experiments, a reduced setting corresponds to a combination of four factors: (1) number of channels for CNNs (c), (2) resolution of input images (r), (3) training epochs (e), and (4) sample ratio (s). The degree of reduction of c, r, and s is represented by a subscript 0, 1, 2, 3, 4. Specific values for the subscript of each reduction factor, for CIFAR-10 dataset [15], are shown in Table 1. The subscript for e is an integer, e.g., e 30 indicates 30 training epochs. The value of c refers to the number of channels in the initial convolutional cell. As for the sample ratio s, we randomly select a subset from the training data according to the value of s and then train the network on this subset. For CIFAR-10, (c 0 , r 0 , s 0 , e 600 ) corresponds to the original or conventional setting [20], which uses 36 initial channels and 32 × 32 input image. All training samples are used for training with 600 epochs. In comparison, (c a , r b , s c , e x ) corresponds to a reduced setting requiring around 1/2 a+b+c FLOPs, i.e., 2 a+b+c speed-up, when compared with (c 0 , r 0 , s 0 , e x ).

Evaluation Metric
We use Spearman Coefficient, denoted as ρ sp , as the metric to evaluate the reliability of reduced settings. The same metric is also used in [18] but it is employed for evaluating the performance of accuracy predictors. In our work, Table 1. Specific values of reduction factors for CIFAR-10 [15]. For the values of cx and rx, if c 0 (r 0 ) 2) x (x is the subscript) is an integer, the integer is taken directly (c2, r2). Otherwise, the nearest number divisible by 4 is selected.  we use the metric to measure the dependence of ranks between the original setting and the reduced settings when the reduced settings are used for ranking the models in the model zoo. Assuming the model zoo has K networks, the formulation of ρ sp is where d i is the difference for network i between its rank in the original setting and its rank in the reduced setting. A higher Spearman Coefficient corresponds to a more reliable reduced setting. An architecture found in a reliable reduced setting is more likely to remain high rank in the original setting. For example, Fig. 3 shows two different reduced settings, i.e., one with a small ρ sp ( Fig. 3(a)) and another one with a large ρ sp (Fig. 3(b)) . When ρ sp is large, the dependence of ranks increases between the original setting and the reduced setting for the same model. This indicates that the average change of ranks when switching from the reduced setting to the original one is small; thus, the reduced setting is more consistent and reliable. When ρ sp is small, the rank obtained from the reduced setting is less reliable. The models with low ranks and high ranks at the original setting could rank highly and low at the reduced setting more frequently, respectively.

Experimental Observations
We try different combinations of reduction factors and investigate the rank consistency between the original and reduced settings by Spearman Coefficient on the CIFAR- 10 dataset [15]. The experimental setting is described in the Appendix. Two useful observations are obtained as follows: 1) With the same iteration numbers, using more training samples with fewer training epochs is more consistent than using more training epochs and fewer training samples. We first analyze the influence of sample ratio s and training epoch e. There are 25 combinations of c and r for each combination of s and e as indicated in Table 1.
As shown in Fig. 4, Spearman Coefficient ρ sp increases when the number of epochs and sample ratio increases. Therefore, the rank consistency improves with more training epochs and/or more training samples. The increase of ρ sp from 30 epochs to 60 epochs is the most obvious, after which the benefits brought about by more epochs become less apparent. When comparing the reduced setting pairs that have the same number of iterations such as c x r y s 0 e z and c x r y s 1 e 2z where x, y ∈ {0, 1, 2, 3, 4}, z ∈ {30, 60}), the results show that training with less epochs but using more samples in each epoch is a better choice than training with more epochs but fewer samples, especially when training for more iterations. Considering the trade-off between computation consumption and benefits, we find the combination s 0 e 60 to be a more optimal setting than other combinations.
2) Reducing the resolution of input images is sometimes feasible while reducing the channels of networks is more reliable than reducing the resolution. We further analyze the effect of reduction factors c and r with fixed s 0 e 60 as discovered in our previous experiments. The 3D-Bar chart in Fig. 5 shows the changes in ρ sp along the dimensions c (X-axis) and r (Y -axis). The Z-axis represents ρ sp for each c x r y setting, illustrated by the height of the bars. We use a measurement called entropy denoted by ρ e to indicate the monotonicity of ρ sp along a particular dimension of reduction factor. The details of this measurement are provided in the Appendix. The entropy ranges from -1 to 1. The absolute value of entropy close to 1 indicates that along this dimension, the objective set has an obvious monotonous increasing or decreasing trend, with only few small fluctuations. On the contrary, if the absolute value of entropy is relatively small, e.g., less than 0.5, then the monotonic increasing or decreasing trend along this dimension is less apparent.
As shown by the values of entropy close to the Y -axis in Fig. 5, the trend along the reduction factor c is obvious. For most of the fixed r settings, fewer channels lead to a betterbehaved reduced setting, except for few points e.g., c 2 r 2 . The variation of entropy close to the X-axis indicates that the trends along r for the fixed c are not apparent. But for most of the cases, a smaller r will lead to a smaller ρ sp , which indicates that reducing the resolution is not reliable. c4r1s0e120 c4r1s0e60 c4r3s0e120 c4r4s0e90 c4r4s0e60 Figure 8. The rank consistency and acceleration ratios of different reduced settings on CIFAR-10 [15]. The X-axis represents the ratio of FLOPs at the original setting to that at the reduced setting.
The Y -axis provides the corresponding ρsp (Eq. (1)). The reduced settings are getting more efficient along X-axis and getting more consistent in ranking different network candidates along Y -axis.
The blue points show all the reduced settings, the orange points are good ones, and the green point is the recommended one.
resolution leads to smaller ρ sp , which is also consistent with the results in Fig. 5. Both Fig. 6(a) and Fig. 6(b) show that more epochs improves ρ sp , which also validates the results in Fig. 4.
The results in Fig. 5 and Fig. 6(a) indicate that decreasing the number of channels will lead to an increase of ρ sp , which means that more reliable setting can be achieved by fewer channels with less computation and memory. Fig. 7 also illustrates this phenomenon, where smaller accuracy difference appears with fewer channels between training and testing dataset. This phenomenon may be caused by overfitting when the amount of parameters for the same architecture is reduced. We hypothesize that overfitting has an adverse effect on the rank consistency.

Economical Evolutionary-Based NAS
Based on the investigations in the previous section, we propose Economical evolutionary-based NAS (EcoNAS), an accelerated version of the evolutionary-based state of the art [23]. Existing evolutionary-based NAS methods [2,19,23,24] suffer from large computation cost due to time wasted on unpromising networks. Taking heed of prior observations (Sec. 3.3), we designed a more efficient and consistent proxy (Sec. 4.1) that reduces the search cost by a large margin. Then we propose a hierachical proxy strategy that trains networks with different proxies based on their respective accuracy. This further improves the search efficiency by focusing more on good architectures and using the most accurate ones for evolution. The search strategy and algorithm pipeline are described in Sec. 4.2.

Efficient Proxy
To replace the original proxies in previous NAS methods [23,35] with a more efficient one, we further analyze the acceleration ratio for FLOPs and ρ sp of the 200 reduced settings and provide empirical results on the good reduced settings. We divide the acceleration ratios into different groups and list the reduced settings that perform best in each group. We also compare two reduced settings in previous works [23,35] where the stack number N is set to 2. In NASNet [35] the reduced setting is c 0 r 0 s 0 e 20 , and in AmoebaNet [23] the setting is c 1 r 0 s 0 e 25 . These two reduced settings exhibit less consistency as shown in Fig. 8. Figure 8 suggests that a proxy that substantially reduces the computation overhead does not necessarily have a poor rank consistency. There are many good reduced settings that possess large acceleration ratio and high consistency at the same time, just as the orange points in Fig. 8. The orange points in Fig. 8 show that many recommended reduced settings include c 4 , e 90 , e 120 . The observation verifies our previous conclusion about the benefits of more epochs and fewer channels. We adopt the reduced setting c 4 r 4 s 0 e 60 in EcoNAS because this reduced setting exhibits relatively higher consistency and acceleration ratios as shown by the green point in Fig. 8. These reduced settings' exact values are shown in Table 1. Under this reduced setting, we enlarge the batch size from 96 to 384 to further compress the searching time.

Hierarchical Proxy
Although we found an efficient and consistent proxy, we observe that training and evaluating each architecture with the same proxy still hurts the search efficiency because unpromising networks count a lot and waste most of the search time. An intuitive approach to this problem is to reduce the training epochs further based on our newly designed consistent proxy. This strategy could find and discard inaccurate architectures faster, however, the best model found from such a proxy with less training epochs might not be as good because the consistency will degrade given fewer training epochs. Therefore, we propose hierarchical proxy strategy as a trade-off to precisely rank good architectures and save time from assessing unpromising networks.
Population set. We divide the population set into three subsets P 1 , P 2 , P 3 , which contains networks with low, middle, and high accuracy and has small, middle, and high probability of being chosen for mutation, respectively. For networks in the sets from P 1 to P 3 , the networks are trained with faster but less consistent proxies (based on observations in Sec. 3.3). We design these three proxies sharing the same c, r, s but training networks with 3E, 2E, and E epochs for P 3 , P 2 , and P 1 (e.g., 90, 60, and 30 if E = 30), respectively. We apply this design because it exploits the weight sharing strategy and it is simple. For example, when several newly mutated networks remain after training E epochs, we only need to train them with E more epochs to rank them more precisely rather than to train with a new proxy from scratch. More diverse proxies for the hierarchical proxy strategy could also be tried and we leave it to future re- The population P k denotes networks trained for k epochs 3: T rain(model, a, b) 4: The function that trains model, starting from epoch a, for totally b − a epochs 5 During each evolution cycle, we use proxy with the lowest computation cost to quickly remove the less accurate networks from newly mutated networks. Then the remaining networks, which are more accurate, will be evaluated with slower but more consistent proxy and assigned into the subsets P 2 , P 3 hierarchically. When choosing networks for mutation from the same subsets, the networks with a higher rank are more likely to be chosen to increase the probability of producing new accurate architectures. Such a design saves the computation overhead by only training those networks with low accuracies using less epochs. Furthermore, allocating resources to more promising architectures allows the good ones to be found with higher precsion according to the observation in Section 3.3. Assigning larger probabilities to these networks for mutation also potentially helps to produce more promising architectures in fewer evolution cycles.
Algorithm pipeline. The pipeline of EcoNAS is shown in Algorithm 1. The initial architectures are randomly generated. All the searched architectures and their accuracies will be stored in the history set. As explained before, in EcoNAS, there are three population sets P 1 , P 2 , and P 3 , also noted as P E , P 2E , P 3E because they store architectures that have been trained for E, 2E, 3E epochs, respectively. For each evolution cycle, the following three steps are conducted: 1. A batch of randomly sampled architectures from the population sets P E , P 2E , and P 3E will be mutated (Algorithm 1, line [12][13][14][15][16][17]. Architectures with higher performance are more likely to be chosen. We follow the mutation rules in AmoebaNet [23] except that we remove the 'identity' mutation, because in EcoNAS the amount of networks at each cycle is relatively fewer. The mutated networks are trained from scratch for E epochs and then added to set P E .
2. Choose the top architectures from set P E and P 2E to P 2E and P 3E , respectively (Algorithm 1, line [18][19][20][21][22][23][24][25]. These architectures are loaded from checkpoints and trained for E more epochs, then those top architectures are more precisely ranked and moved to the corresponding subsets.
3. Remove dead architectures from set P E , P 2E and P 3E to force the algorithm to focus on newer architectures [23].

Experiments
Implementation Details. For EcoNAS, the number of nodes (Fig. 2) within one architecture is four and we set the stack number N = 6 to build the networks. We found in our experiments that N = 2, a setting used in previous works [23,35] is detrimental to the rank consistency (Fig. 8). Our searching experiments are conducted on CIFAR-10 [15] and we split 5k images from the training set as validation set. The search space is similar to that in [20]. Initially, P 1 has 50 randomly generated architectures, while P 2 and P 3 are empty. At each cycle, 16 architectures sampled from P k , k ∈ [E, 2E, 3E] will be mutated. We set the number of cycles C to 100 and training epoch E to 20. All the experiments and search cost calculation are conducted using NVIDIA GTX 1080Ti GPUs.

Overall Results on CIFAR-10 and ImageNet
The searching process yields several candidate convolutional cells and these cells are evaluated on two datasets, CIFAR-10 [15] and ImageNet [25] for image classification tasks. Unlike previous works [23,33], which select dozens or hundreds of cells to evaluate, we only pick up the top-5 cells from history to evaluate the performance of our search algorithm. Retraining top-5 cells saves the retraining overhead and is enough in our experiments; the reason is analyzed in Section 5.3.
Results on CIFAR-10. For the task of CIFAR-10, we set N = 6 and the initial channel c = 36. The networks are trained from scratch for 600 epochs with batch size 96. Other hyper-parameters follow previous works [20,30]. The experimental results on CIFAR-10 are shown in Table  2. The best model found by our algorithm achieves a test error of 2.60%, a rate that is on par with state-of-the-art  [20,30] and weight sharing method [22].
Results on ImageNet. To evaluate the transferability of the cells searched by EcoNAS, we transfer the architectures to ImageNet, where we only change the stack number N to 14 and enlarge the initial channel c to 48. As shown in Table 3, the best architecture found by EcoNAS on CIFAR-10 generalizes well to ImageNet. EcoNAS achieves top-1 error that outperforms the previous works that consume the same magnitude of GPUs [20,30]. EcoNAS also surpasses models with similar amount of parameters (i.e., having fewer than 6M parameters) found by reinforcement learning [35] and evolution algorithms [23], which require about 200× more GPU resources.

Ablation Study of EcoNAS
Reduced setting for NAS using evolutionary algorithms.
We evaluate the reduced setting on the conventional evolutionary algorithms in Table 4. AmoebaNet [23], NAS-Net [35], and our EcoNAS use the same search space but they have different in search algorithm design. By simply replacing the reduced setting of AmoebaNet [23] to more consistent ones, its computational costs are remarkably reduced and the accuracies of searched models also increase, with and without applying hierarchical proxy strategy.
Reduced setting for other NAS methods. We further evaluate the effectiveness of the newly discovered reduced settings on other NAS methods, e.g., gradient-based methods [5,20]. We report the top-1 error rates of DARTS [20] on CIFAR-10 [15] and ProxylessNAS [5] on ImageNet-1k [25], as shown in Table 5. Directly reducing the channels and the input resolutions by half accelerates these methods and finds models that keep comparable accuracy in comparison to those searched by the original proxies. This validates our observation reported in Sec. 3.3 and the effectiveness of our newly designed reduced setting.
Hierarchical proxy strategy. We evaluate the hierarchical proxy strategy on the conventional evolutionary method [23] in Table 4. The reduced settings c 4 r 4 s 0 e 60 and c 4 r 4 s 0 e 35 train the same number of models (1k) for the same number of epochs (35k) with or without the hierarchical proxy strategy, respectively. Both the search cost and error rate are reduced after applying the hierarchical proxy strategy. The results suggest the search efficiency of hierarchical proxy strategy and its effectiveness in finding better networks using less search time. Number of Retained Good Models top-10 models among 20 models top-10 models among 15 models Figure 9. Ability to retain top models for reduced settings with different ρsp.

Analysis
Ability of reliable proxies to retain top models. We further evaluate the ability of the reduced settings in retaining good models. If a top-10 model at the original setting is a top-15 or top-20 model at the reduced setting, it will be considered as a 'good model retained'. We divide the reduced settings into different groups according to their corresponding ρ sp and calculate the average number of good models retained for reduced settings in the same group. As ρ sp increases, the number of good models retained also increases ( Fig. 9), suggesting that reduced settings with better consistency can keep more good models in the top positions. Figure 9 also indicates that reliable proxies can reduce not only the search cost but also the retraining overhead. Previous works need to select the best model after retraining hundreds of top models searched under proxies (e.g., 100 [33] for BlockQNN and 250 for NASNet [35]), Amoe-baNet [23] also needs to retrain 20 models. One of the reasons for this phenomenon is that previous works adopt less reliable proxies as shown in Fig. 8. Since the ranks of networks under the proxies are less consistent with their actual ranks under the original setting, they need to select more networks for retraining to find the optimal one. According to the conclusion obtained from Fig. 9, a more consistent reduced setting retains more top models, thus allows the search algorithm to retrain fewer networks to obtain competitive accuracy. With a more reliable proxy, EcoNAS only retrains top-5 networks and remarkably saves computation overhead for retraining models, which is usually overlooked in most NAS literature.
Diversity of structures by hierarchical training strategy. EcoNAS adopts hierachical training strategy to assign newly mutated networks into three population sets and sample networks for mutation from these sets with different probabilities. Under this scheme, both the good and bad models enjoy chances to mutate so that the architectures in the next evolution cycle will not be trapped in a few local optimal structures. We evaluate the average accuracy of networks in the three poulation sets during evolution cycles as shown in Fig. 10. The difference of average accuracy for networks in P E , P 2E , and P 3E are apparent and do not deviate much during evolution cycles, but the average ac- curacy for networks in those sets increase gradually, which verifies the diversity of structures in those population sets. The diversity of structures allows the search algorithms to find accurate architectures with fewer search costs by potentially helping to explore the search space more comprehensively. Since hierarchical training strategy provides the diversity of structures in the population sets, EcoNAS obtains similar competitive architectures after evaluating 1k models. As a comparison, BlockQNN [33] evaluates 11k models, AmoebaNet [23] evaluates 20k models, and NAS-Net [35] evaluates 45k models.

Conclusion
In this work, we systematically study the behaviors of different reduced settings on maintaining the rank consistency in Neural Architecture Search. By conducting extensive experiments on different combinations of reduction factors, we observe that (1) with the same iteration numbers, using more training samples with fewer training epochs is more consistent than using more training epochs and fewer training samples; (2) reducing the resolution of input images is sometimes feasible while reducing the channels of networks is more reliable than reducing the resolution.
We also propose Economical evolutionary-based NAS (EcoNAS) that can reduce the search time by about 400× in comparison to the evolutionary-based state of the art [23]. In EcoNAS, we first design a new, fast, and consistent proxy to accelerate the search process based on the aforementioned observations, which also reduces the retraining overhead by retaining more top models. Then we propose to use a hierarchical proxy strategy to assess architectures with different proxy based on their accuracy. This new strategy improves search efficiency and is capable of finding accurate architectures with less search overhead while exploring the search space more comprehensively.
Last but not least, we find some proxies led by our obeservations are also applicable to other NAS methods [5,20]. These proxies further reduce the search time while the discovered models achieve comparable or better performance. We wish our work could inspire the community to further explore more practical proxies and search algorithms to improve the efficiency of NAS.

A1. Two Model Examples in Introduction
In the introduction, we mentioned that some architectures applied with certain reduction factors win in the reduced setting but perform worse in the original setup on CIFAR-10 [15]. The normal and reduction cells for the aforementioned models A and B are shown in Fig. A1 and Fig. A2, respectively. The results on CIFAR-10 of the two models in the original setting and reduced setting are shown in Table A1. The original setting is c 0 r 0 s 0 e 600 while the reduced setting is c 0 r 0 s 0 e 30 . The training details about the two settings are provided in Section A4.2. The results show that the rank of performance evaluated in the reduced setting is not guaranteed to be the same as that in the original setting.

A2. Construction of Model Zoo
This section provides the details on constructing the model zoo (Section 3). Each network architecture in the model zoo is a stack of normal cells alternating with reduction cells. In each network, these two cells are all generated separately according to the common selection steps in [18,22,35] and we just replace the search algorithm in these approaches by random sampling. The number of nodes inside the cells is 5 and every cell receives two initial inputs. For cell k, the two initial inputs are denoted as h k−2 and h k−1 , which are outputs of previous cells k − 2 and k − 1 or the input of images. The output of each cell is the depthwise concatenation of all the intermediate nodes (two initial inputs excluded). The generation steps of each intermediate node are as follows: • Step 1. Randomly select an input from the input set, which contains two initial inputs of the cell and the set of outputs from previous nodes within the cell. • Step 2. Randomly select another input from the same input set as in Step 1.
• Step 3. Randomly select an operation from the operation set and apply this operation to the first input selected in Step 1.
• Step 4. Randomly select another operation to apply to the second input selected in Step 2.
• Step 5. Add the outputs of Step 3 and Step 4 to create the output of the current node.
The original ' Step 5' in [35] provides two combination methods: element-wise addition and depth-wise concatenation. However, previous work [18] mentions that the concatenation method are never chosen during search. Therefore, we only use addition as the combination operation. We selected 13 operations to build our operation set considering their prevalence in the NAS literature [5,20,23,35], which are listed as below:

A3. Detailed Information About Entropy
In Figure 5 and Figure 6 of Section 3.3, a new measurement called entropy is used. This section provides the details on how entropy is calculated.
We use entropy, denoted by ρ e , to measure the monotonicity of a given objective set. The entropy ρ e is the Spearman Coefficient measuring the rank difference between the objective set and an arbitrary increasing collection (called base set, such as {1, 2, 3, 4, 5}). The objective set is the collection of ρ sp along a certain reduction factor dimension, such as ρ sp of reduced settings {c 0 r 0 s 0 e 30 , c 1 r 0 s 0 e 30 , c 2 r 0 s 0 e 30 , c 3 r 0 s 0 e 30 , c 4 r 0 s 0 e 30 } along the dimension of reduction factor c. If the absolute value of ρ e is closer to 1, it indicates that the objective set has a more apparent increasing (ρ e approximates 1) or decreasing (ρ e approximates −1) trend. Otherwise (e.g., ρ e approximates 0) the monotonicity of the objective set is less apparent. Since the true values of the inputs will be transferred to the ranks, the choice of base set will not affect the final results if it is a set of increasing numbers.

A4.1. Implementation Details of EcoNAS
Search space. The search space of EcoNAS consists of 8 operations, which follow the previous work [20] and are listed as follows: Each cell in the network consists of 4 nodes ( Figure 2). The generation of each node follows the 5 steps described in Section A2, except that the operation sets are different. In one cell, the node outputs that are not used will be concatenated together as the cell output [23,35].
Search strategy. We use the setting of c 4 r 4 s 0 and the batch size is 384. Every network is trained on a single GPU. In every cycle, the chosen networks will be trained for 20 epochs and the maximum training length for each network is 60 epochs, i.e., the complete reduce setting is c 4 r 4 s 0 e 60 , which has been found to be effective in the main text. The other hyper-parameters remain the same as stated in Section A4.2. We use P 20 , P 40 and P 60 to denote the networks trained for 20, 40, and 60 epochs, respectively. In each cycle, 16 networks will be chosen from the population and be mutated, and the top-8 and top-4 networks in P 20 and P 40 will continue to be trained for 20 epochs, which means that no more than half of the networks in the P 20 and P 40 set will get chance to be continually trained. When the process is finished, we only retrain and find the best model from top-5 models from P 60 . The searched models that achieve the best and the second best results are shown in Figure A3 and Figure A4, respectively.

A4.2. Implementation Details on CIFAR-10
This section provides the details of training strategies for the original and reduced settings on CIFAR-10 (Section 3.3 and 5.1). In the original setting, we train each network from scratch for 600 epochs with batch size of 96. Cosine learning rate schedule is used with lr max = 0.025 and lr min = 0.001 and the weight decay is 3e−4 [20]. Additional enhancements including cutout [8], path dropout [16], and common data augmentations follow the previous work [20].
The implementation for the reduced setting follows that for the original setting, except those as follows: 1. The number of training epochs is decided by the reduction factor e. But the cosine learning rate scheduler still finishes a completed cosine cycle within the reduced epochs. 2. Path dropout is excluded in the reduced setting because we empirically find that the evaluation ability of reduction settings will increase if path dropout is excluded. The possible reason is that we use very small number of epochs, which is not favored by path dropout. 3. The images are resized to reduced resolution after padding and random cropping, and the cutout length is adjusted according to the reduced resolution.  Figure A3. Normal and reduction cell structures of first-place model, whose error rate is 2.60% on CIFAR-10.  Figure A4. Normal and reduction cell structures of second-place model, whose error rate is 2.67% on CIFAR-10.