{'fraction_non_alphanumeric': 0.02380368098159509, 'fraction_numerical': 0.01736196319018405, 'mean_word_length': 4.870003601008282, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NAAP-440 Dataset and Baseline for Neural Architecture Accuracy Prediction

Neural architecture search (NAS) has become a common approach to developing and discovering new neural architectures for different target platforms and purposes. However, scanning the search space is comprised of long training processes of many candidate architectures, which is costly in terms of computational resources and time. Regression algorithms are a common tool to predicting a candidate architecture's accuracy, which can dramatically accelerate the search procedure. We aim at proposing a new baseline that will support the development of regression algorithms that can predict an architecture's accuracy just from its scheme, or by only training it for a minimal number of epochs. Therefore, we introduce the NAAP-440 dataset of 440 neural architectures, which were trained on CIFAR10 using a fixed recipe. Our experiments indicate that by using off-the-shelf regression algorithms and running up to 10% of the training process, not only is it possible to predict an architecture's accuracy rather precisely, but that the values predicted for the architectures also maintain their accuracy order with a minimal number of monotonicity violations. This approach may serve as a powerful tool for accelerating NAS-based studies and thus dramatically increase their efficiency. The dataset and code used in the study have been made public.

The main drawback of the NAS approach is its resource consumption, which for instance, could be 500 GPUs running for 4 days [34]. While it is a rather trivial and inexpensive task to evaluate a candidate architecture in terms of efficiency, by counting MACs or measuring its runtime on a single batch [23][24][25], it is a rather expensive task to compute the architecture's accuracy, as it requires training it first using a given recipe.
We aim at introducing a new baseline for developing regression algorithms and a set of features that will allow for correctly predicting a candidate architecture's accuracy with a minimal cost of resources. By saving 90% up to 100% of an architecture's training time, we anticipate that our approach will join the efforts to make NAS dramatically more efficient, which will allow for the scan of wider search spaces and thus the discovery of better architectures.
For that purpose, we introduce the NAAP-440 dataset, which consists of 440 neural architectures trained and evaluated on the CIFAR10 dataset [10]. As visualized in Figure 1, we generate 440 neural architectures using a set of constraints and use a fixed training recipe to train each of them. After each epoch, we document the value of the loss function and evaluate the model on the CIFAR10 test set. As a result, our dataset contains both scheme features and features from the training process. We evenly sample 40 architectures, to form a training set of 400 architectures and a test set of 40 architectures. Our aim is that, given a minimal number of features from the training epochs, to predict an architecture's highest accuracy score achieved on the CI-FAR10 test set, over all its training epochs. We perform ablation studies on the feature set and provide a baseline of performance reports using many regression algorithms and combinations of features. We measure the quality of the accuracy prediction using two measures: mean absolute error and monotonicity score.
Our experiments indicate that by using off-the-shelf regression algorithms, it is actually possible not to train the architectures on CIFAR10 at all, or in other words, to accelerate the training process by 100%, and still predict their CIFAR10 test set accuracy scores with MAE of 0.5% and only 42 monotonicity violations, out of 40 2 = 780 possible violations. Furthermore, the experiments indicate that by only training each architecture on CIFAR10 for 10% of the epochs and evaluating them after every epoch, or in other words, accelerating the training process by 90%, it is possible to predict their CIFAR10 test set accuracy scores with MAE of 0.4% and only 25 monotonicity violations, as demonstrated in Figures 5,6 and Table 3. We believe that using this approach, it will be possible to accelerate any NAS-based solution, by building similar regression algorithms using a rather small population of architectures sampled from the search space, then using the regression algorithms for the whole search space.
Our contributions in this study are as follows. First, we create the NAAP-440 dataset and make it public for further research. Second, we conduct ablation studies and experiments that form a baseline of performances to be used as a reference. Third, we introduce the monotonicity-based evaluation and show that evaluating candidate architectures can be accelerated by up to 100% and still be accurate.
We encourage researchers to use the NAAP-440 dataset in their studies and try to improve it from any aspect, including improving the reported baseline results, extending the evaluation metrics and extending the dataset.

Dataset
Scheme Generation. The schemes are generated using a DFS scan on the convolutional layers with a set of constraints. As visualized in Figure 2, some constraints are imposed on the whole model, while others are imposed per layer. For the model itself, the number of convolutional layers varies between 3 and 4 and the number of stages varies between 2 and 3. The layers themselves can vary in their kernel size, width, stride and whether they have a skip connection. Every convolutional layer is followed by batch normalization [8] and ReLU. We always use convolutional layers with no bias and with padding of S 2 , where S is the kernel size. Since layers 1,2 are fixed to have stride=2, only up to one of layers 3,4 can be assigned with stride=2, to enforce the number of stages varying between 2 and 3. A skip connection can be applied only if the layer's input shape agrees with its output shape, which can only occur when the stride is selected to be 1 and the width of the current layer is equal to the width of the previous layer. The total number of valid schemes according to our set of constraints is 440.
Architecture Training. In order to generate the dataset, we turn the 440 schemes into actual architectures and train each of them on the CIFAR10 dataset for 90 epochs. For reproducibility, we initialize a fixed random-seed before initializing each model's weights and before initializing the shuffled training set loader, as well as switching the accelerators to deterministic mode. This ensures that all the architectures are trained using the same permutation of the training set. We use CE loss and the SGD optimizer with warm restarts [14] every 3 epochs. The training recipe is as follows: batch-size: 256, momentum: 0.9, weight-decay: 0.0001, learning-rate: 0.1, with exponential decay, by multiplying by 0.1 after every epoch. As the whole training runs for 90 epochs and the learning rate restarts after every 3 epochs, the process can be described as a sequence of 30 training cycles, each consisting of 3 epochs.
Dataset Structure. We build a tabular dataset that contains all the 440 architectures. Each architecture is reported with its scheme details and the collected data from its training and evaluation on CIFAR10. The scheme details include the network's depth, number of stages, first and last layer widths, number of total parameters and number of total MACs. The data from the training and evaluation on CIFAR10 is provided for each of the 90 epochs and includes the model's accuracy on CIFAR10 test set after each epoch is completed, as well as each epoch's mean and median training loss over all the SGD batches.  Separating into Train and Test Sets. We would like to create a fair separation of the 440 architecture data records into train and test sets. Our first choice deals with quantities -we would like to allocate 400 samples to the training set and 40 samples to the test set. The second and more complicated choice is how to select the 40 test samples in a deterministic and even way, in terms of the network structure and performances. The original order of the architectures has been determined by the DFS scan over the set of constraints, which we consider unsatisfactory. Instead, as visualized in Figure 3, we sort the architectures by their accuracy scores on the CIFAR10 test set, using the highest achieved score over all the training epochs. Then, we arrange the 440 architectures in 40 bins, such that each bin consists of 11 architectures that have similar accuracy scores on the CIFAR10 test set. From each bin, we allocate the architecture at the central index to the test set. This deterministic operation leaves us with 400 training samples and 40 representative test samples.

Evaluation Metrics
Prediction Accuracy. In order to evaluate the quality of a proposed regression model, we use multiple measures. The first and rather obvious one is the Mean Absolute Error (MAE) metric, which reflects the overall quality of the regression model.
Monotonicity Score. Being able to compare two candidate architectures to each other is an important capability when searching for neural architectures. Using a regression model's predicted accuracy of the two architectures can serve as a measure for comparison. However, while some proposed regression models might provide accurate predictions for most of the test samples, their predictions are not guaranteed to preserve the original order of the samples, which may lead to a misleading comparison. For that reason, another metric that we propose is a monotonicity score. As visualized in Figure 4, given N samples, the maximum possible number of monotonicity violations, is the number of pairs, which is N 2 = N (N −1)/2. Therefore, the monotonicity score we propose is while in our case, N = 40, such that N 2 = 780. Thus, the monotonicity score will be equal to 1 when the predictions maintain the order of the architecture accuracies perfectly. In contrast, it will be equal to 0 when the predictions result in an exactly reversed version of the original order of the architecture accuracies, a situation where every pairwise comparison of architectures using the regression model will be incorrect.
Search Acceleration. The architecture accuracy prediction can be performed under different conditions. Since our aim is to predict the highest test accuracy on CIFAR10 in the course of 90 training epochs, a regression model that only uses an architecture's details from the 9 first epochs, which are 10% of the training process, accelerates the search by 90%. Similarly, a regression model that only uses the schematic features accelerates the search by 100%, as it requires no training at all. We therefore evaluate different regression algorithms on the dataset under different feature sets, to emphasize the trade-off between search acceleration, prediction accuracy and prediction monotonicity.

Evaluation Baseline
Scheme Features Ablation Study. As described in Section 2, the dataset's schematic details include 6 features, which are the network's depth, number of stages, first and last layer widths, number of total parameters and number of total MACs. We aim at checking whether all of the scheme   Table 3. Baseline results using various regression algorithms. For linear regression with a specified degree D, the model is trained to predict y = ( w · x + 1) D rather than y = w · x. It is achieved by training the model to predict y 1/D − 1. All ensemble regressors are specified with their number of estimators, N.
features are essential to training a regression model. We were surprised to witness most of the scheme features actually harming the regression accuracy repeatedly, under different regression algorithms, including Random Forest, Gradient Boosting and Linear Regression. Table 1 demonstrates that that number of parameters and number of stages are most essential for an accurate regression, as removing them from the feature set causes a notable degradation. On the other hand, the removal of each of the other features actually contributes to the regressor performance, when using a Random Forest regressor. However, when we do combine the features from the training process, which as described in Section 2, include each epoch's accuracy on CIFAR10 test set as well as its mean and median loss on the training set, we observe that the complete set of scheme features is essential. Table 2 shows the essence of the complete set of scheme features when features from the training process are included. Furthermore, when using a linear regression model, we find it very essential to use a logarithmic version of the number-of-parameters feature. When only using scheme features, it reduces the trained regression model's number of monotonicity violations from 89 too 64 and the MAE from 2.3% to 1.7%. To conclude, we recommend to apply the logarithmic function on the number-of-parameters feature, to only use the limited set of scheme features when choosing not to train the model at all and to keep using all 6 scheme features when including features from the training process.
Baseline. After we have understood the effect and contribution of each feature to the performance of the regressor, we proceed to providing a baseline using many regression algorithms under different settings in Table 3. We used the scikit-learn [17] Python package for that purpose.
Since some of the regression algorithms have different performances under different random seeds, we repeat running each algorithm 5 times using different seeds and report their median performance, according to the number of violations. As guided by the ablation studies, we use the limited scheme feature set when not using features from the training epochs, while using all the scheme features when we do use features from the training epochs. Furthermore, we exclude results when using features of more than 9 training epochs, since as demonstrated in Table 2, we observe that features from further training epochs usually do not contribute to and even harm the performance. Figures 5,6 demonstrate the strengths of different regression algorithms with different feature sets. In all the experiments conducted, all features are fed to the regressors after scale and bias normalization, which is learned from the training set.
Naive Reference. As a reference for the regression algorithms, we report that the MAE achieved by a dummy regressor, which always outputs the mean of the training set, is 0.043. As all architectures are predicted with the same accuracy, their prediction-based order is equivalent to any random permutation. It should be possible to prove that the average number of violations over all permutations is identically 1 2 N 2 . We check 500 random permutations of the 40 architectures, and as we expect, we observe that their average number of monotonicity violations is 390 = 1 2 40 2 , which results in a monotonicity score of exactly 0.5. Therefore, a reasonable regressor should achieve MAE < 0.043 and less than 390 monotonicity violations on the NAAP-440 test set. The naive reference emphasizes the importance of using the monotonicity score alongside the MAE metric.

Conclusions and Future Work
We have presented the NAAP-440 dataset for architecture accuracy prediction, which includes 440 schemegenerated neural architectures trained and evaluated on the CIFAR10 dataset. Our experiments show that by accelerating the training process by up to 100%, it is possible to predict an architecture's accuracy score quite precisely, as well as maintain the 40 test architecture's accuracy-based order with a quite a low number of violations. As we have published the dataset and the code, we hope that the dataset will be used by other researchers to improve and push the boundaries of neural architecture accuracy prediction capabilities.
For future work, we may consider (1) examining additional train/test divisions that may require the usage of regression algorithms that can extrapolate, (2) enriching the dataset with more architectures, (3) creating another dataset that will have a fixed architecture and variable training recipes, as opposed to this work's fixed training recipe and variable architectures.