{'fraction_non_alphanumeric': 0.03490317953621803, 'fraction_numerical': 0.011206072196987807, 'mean_word_length': 4.850524475524476, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SuperNet in Neural Architecture Search: A Taxonomic Survey

Deep Neural Networks (DNN) have made significant progress in a wide range of visual recognition tasks such as image classification, object detection, and semantic segmentation. The evolution of convolutional architectures has led to better performance by incurring expensive computational costs. In addition, network design has become a difficult task, which is labor-intensive and requires a high level of domain knowledge. To mitigate such issues, there have been studies for a variety of neural architecture search methods that automatically search for optimal architectures, achieving models with impressive performance that outperform human-designed counterparts. This survey aims to provide an overview of existing works in this field of research and specifically focus on the supernet optimization that builds a neural network that assembles all the architectures as its sub models by using weight sharing. We aim to accomplish that by categorizing supernet optimization by proposing them as solutions to the common challenges found in the literature: data-side optimization, poor rank correlation alleviation, and transferable NAS for a number of deployment scenarios.


Introduction
Designing neural architectures has been essential in a variety of visual recognition tasks. In general, such architectures are being hand-engineered using expert knowledge and heuristics He et al., 2016;Howard et al., 2017] due to vanishing/exploding gradient issues. ResNet [He et al., 2016] utilizes skip connections where outputs of a layer jump over to a deeper layer to alleviate such issues while there still remain labor-intensive jobs that require expert knowledge to deploy them into real-world problems. Neural Architecture Search (NAS) lessens such issue by automating the process of finding such architectures. During search time, the optimal architecture is identified with certain evaluation criterion.
With the rapid growth of interest in NAS, there has been a lot of recent literature based on Reinforcement Learning (RL) Baker et al., 2016], Evolution-ary Search (ES) [Real et al., 2017;Elsken et al., 2018] and Bayesian Optimization (BO) [Shi et al., 2019;Eriksson et al., 2021; despite their expensive time costs for searching processes. Early RL and ES studies can require repetitive trial-and-error that results in over thousands of GPU days for search time because both approaches train numerous architectures by directly sampling the candidates from a large search space. As a result, this makes such NAS methods infeasible for most users.
The supernet approach lightens such computational burden [Pham et al., 2018]. A supernet is a network that contains all possible architectures for the search process. As the name suggests, supernet-based methods first train a supernet which embodies the entire search space. The method then takes a subnet of the supernet as the sampled architecture. Unlike earlier approaches that requires repetitive training, supernetbased NAS eliminates that requirement. This approach reduces inference time from thousands of GPU days by orders of magnitude. We refer to the optimization problem implicit in supernet approach as supernet optimization, drawing a connection (and contrast) to NAS optimization.
Due to its wide applicability, supernet optimization works have been numerous in the NAS literature [Pham et al., 2018;Bender et al., 2018;. Considering how supernets reduce the search time drastically, many applications can be found in real-world scenarios such as edge devices (e.g., smart phones, internet of things) [Cai et al., 2018]. Beyond lessening computation cost, supernet based methods have other benefits: CO 2 emission reduction [Cai et al., 2019], better model transferability across different datasets [Zoph et al., 2018] and tasks [Pasunuru and Bansal, 2019;Gao et al., 2020]. Running machines produce CO 2 , so cutting down the train and search cost would reduce the emission levels. Model transferability would also reduce the train and search cost because it would omit the need for searching a new architecture for every hardware setting, thus leading to lower CO 2 emission and faster deployment. However, despite its popularity in NAS research, to the best of our knowledge, there is no recent survey on supernet optimization, and thus we aim to provide insights for the supernet literature.
This paper categorizes supernet optimization with three common problems found throughout the literature (Figure 1): • Data-Side Issue. A common avenue to optimize a machine learning algorithm is to address its data-related  Figure 1: A visualization of problems found in supernet optimization literature. On the upper left, we provide a diagram of a supernet cell where the circles represent feature maps and rectangles denote candidate operations (e.g. convolution, pooling, skip connection). In the upper right, we illustrate how a supernet can be constructed by connecting repeating cell modules. In the lower half, we illustrated the three challenges we present in this survey. We discuss multi-resolution, class noise, and out-of-distribution for data-side issues. Multi-resolution refers to the case where the images are of various sizes. Class noise refers to the presence of mislabelled data in the supervised case. Out-ofdistribution refers to the mismatch of training data between the training and live stage. Poor rank correlation is when sub-networks sampled from a supernet perform significantly worse than standalone accuracy. The line labelled as ideal refers to perfect correlation. I.e., the predicted accuracy coincides with the standalone accuracy. For transferability, specialized networks that require no extra training can be sampled from a large network and be directly deployed to specific hardware settings.
issues. In image classification, the resolution size is of concern. Most supernet optimization methods are trained to perform on ImageNet for 224 × 224 pixels [Zhong et al., 2018;Xie et al., 2018;Chu et al., 2021;Cai et al., 2019;Yu et al., 2020;Wang et al., 2021b;. Given the prevalence of high resolution images in real life settings, research on supernet optimization methods that can handle such images are of interest. Another issue is data labelling. Supervised learning requires ground truth labels which are laborious to provide. In addition, most machine learning algorithms are built under the i.i.d. assumption (independent identically distributed). Experiments in the deep learning literature usually deal with clean data using CIFAR10/100 and ImageNet. However, in real life settings, there can be noise in the input data and labels. This can hinder model performance for supernet optimization. Such cases are referred to as Out-of-Distribution (OoD). Thus, there is a need for developing NAS methods that are robust to OoD data.
• Poor Rank Correlation Issue. When a supernet-based NAS method yields an architecture, it evaluates the sampled network with a proxy performance metric. It repeats this process and outputs of a list of architectures ranked by their predicted performance. However, the estimated accuracies have low correlation with the standalone accuracies. Many works address this issue [Bender et al., 2018;Chu et al., 2021;]. • Transferability. Transferability for a number of deployment scenarios across different hardware settings is of interest for applications of NAS. Despite its increasing demand, many works do not take hardware setting transferability into consideration during the search process Real et al., 2017;Bender et al., 2018]. Supernets are no exception to high compute cost because they require retraining after sampling a sub network Bender et al., 2018]. To accommodate for all different hardware settings, users would have to repeat the costheavy process repeatedly for each setting. Organization. This survey is structured as follows. In Section 2, we give preliminaries to provide necessary background information for supernet optimization. In Section 3, we go over the optimization strategies that tackle the aforementioned three issues in supernet optimization. In Section 4, we compare the performances of different architectures found by human experts and NAS methods. In Section 5, we provide insights on potential future directions for supernet-based NAS for image classification. And lastly, we conclude our paper in Section 6.

Neural Architecture Search (NAS)
Neural Architecture Search (NAS) automatically finds an architecture for a deep learning task while satisfying accuracy constraints. The NAS process can be broken down into three parts: the search space, the search strategy, and the performance estimation strategy [Elsken et al., 2019].
• Search space. The search space embodies the possible architectures for the given task. Introducing inductive bias when defining the search space is inevitable. However, works involving meta-learning [Lee et al., 2021] constructs an embedded search space that circumvents such human bias. The earliest example of a search space would be a sequential layer of operations.
• Search strategy. The search strategy determines how the architectures are sampled from the search space. There are two competing metrics here: architecture performance and search time. Finding high performing architectures may take a long time. A potential navigation method for this would be probing the Pareto frontier; the set of solutions where improving one objective will worsen another objective. Reinforcement learning  and evolution [Real et al., 2017] are other examples of search strategies.
• Performance estimation strategy. The performance estimation strategy seeks to measure how well the searched architecture does on new unseen data. The naive way to do this is to predict on the validation set. Of course, full-training is computationally taxing and not the most practical way to do things. Works in the NAS literature has found many different proxies for performance estimation.

Supernet Optimization
We remark that the terms supernet and weight-sharing are at times used interchangeably and circularly in the NAS literature. For the purposes of this survey, we will distinguish and define those terms. A supernet is a neural network that contains all possible architectures to be sampled, i.e., it serves as the search space. Architectures are found by taking subnets from the supernet. We can represent a supernet as a Directed Acyclic Graph (DAG) where the nodes are feature maps and the edges are operations (e.g. convolution, pool, zero). We refer to figure 2 for an illustration of a supernet.
A key feature of supernet-based NAS would be weightsharing [Pham et al., 2018]. Weight-sharing is done by first training a single large supernet. An RL-based controller [Pham et al., 2018] then samples a subnet from the supernet. As a result, all sampled child networks share weights from the supernet. The introduction of weight-sharing has reduced GPU days for the search process from thousands of GPU days down to a few GPU days. Real et al., 2017;. The one-shot approach [Bender et al., 2018] extends the weight-sharing technique. The authors train a single large supernet which is just a neural network trained using SGD with Momentum. During training, a linearly scheduled dropout strategy [Hinton et al., 2012] is used to prevent co-adaptation among operations (edges). The dropout rate starts at 0 and then linearly increases to r 1/k where 0 < r < 1 is a hyperparameter and k is the number of incoming paths to a given operation. When applying dropout, a random subset of the operations are zeroed out. After training, subnets are randomly sampled from the trained supernet following a uniform distribution (other search methods such as evolution or reinforcement learning can be used instead). The sampled subnet is then evaluated on a held-out validation set. The resulting output is a list of candidate architectures ranked by one-shot accuracy (based on validation set). The best performing architectures can be selected to be retrained from scratch.

Optimization Strategies
In this section, we survey supernet optimization methods with respect to the following optimization strategies used to address the aforementioned issues.
• Data-Side Optimization. Corresponding to data-side issues, we survey supernet-based NAS methods that are tailored for high resolution images [Ding et al., 2021]

Data-Side Optimization
A lot of works in supernet optimization focus on improving the architecture or the search phase itself. Works such as OFA [Cai et al., 2019] and BigNAS [Yu et al., 2020] seek to yield architectures that can be instantly deployed for tasks (these are mentioned in section 3.3). Methods such as ENAS [Pham et al., 2018] and DARTS  focus on searching architectures efficiently (quickly). However, the actual architecture and search phase are not the only avenues for improving a machine learning algorithm. Data is another route for optimization. The issue of data is essential to any machine learning task. In this section, we survey supernet-based NAS with respect to data-side optimization. Given that the task is image classification, the image resolution is of concern. We cover High Resolution NAS (HR-NAS) [Ding et al., 2021] which is designed for high resolution images. For supervised learning, labels are integral. However, labelling high volumes of data is laborious and can be a bottleneck in the machine learning pipeline. We explore Random Label NAS (RLNAS)  which removes ground truth labels and only uses random labels for the search. In addition, most NAS methods are fed input signals with clean data. There is a need for NAS methods that are robust against noisy data. We also look into NAS that is robust against Out-of-Distribution (OOD) data .

High Resolution Images
Image resolution sizes in image classification tasks tend to be on the smaller side (about 224 × 224). Naturally, computational requirements increases as image resolution goes up. So a natural concern is to develop methods that handle high resolution images. High Resolution NAS (HR-NAS) searches for architectures designed to take in high resolution images.
HR-NAS introduces a light-weight Vision Transformer (ViT) [Dosovitskiy et al., 2020] that also incorporates convolution. The ViT consists of a projector, an encoder, and a decoder. For the projector, instead of using a sinusoidal encoding, the authors project the input feature with a simple normalized 2D positional map. For the encoder, the input feature is transformed into a set of n tokens where each is a lower-dimensional semantic embeddding with positional information. Then the tokens are fed as queries, keys, and values into the ViT.
The authors introduce a multi-branch search space inspired by HRNet . This search space contains both multi-scale features and global contexts while maintaining high resolution representations throughout the network. The supernet in HR-NAS is a multi-branch network where each branch is a chain of search blocks operating at different resolutions. Each search block combines a MixConv and a Transformer [Vaswani et al., 2017]. The search is based on progressive shrinking that discards some convolutional channels and Transformer queries during training.

Randomly Labeled Data
Random Label NAS (RLNAS)  abandons ground truth labels and adopts random labels during searching. The authors of RLNAS decouple two convergence-based optimization steps.
First, they train a supernet with random labels: where W, α, A, R, L, Γ(A) denote the weights, architecture, architecture space, random labels, loss function, and a uniform distribution of α ∈ A, respectively. Second, the authors run an evolution algorithm that searches the optimal architecture based on a convergencebased metric Convergence(·) as fitness: (2) The random labels follow a uniform distribution and the convergence-based metric is an angle-based model evaluation metric where it takes the angle between initial model weights and trained ones.
Out-of-Distribution Generalization NAS for image classification mostly works with clean datasets such as CIFAR10/100, ImageNet, etc. However, clean data is scarce in real world settings. In addition, data distribution shifts are often found in real world settings. For example, a dog breed identifier might be given a photo of a cat due to user error. Thus, there is a need for supernet optimization to be robust to Out-of-Distribution (OOD) cases. Out-of-Distribution refers to the case where a machine learning algorithm is fed data that is not from the distribution it was trained on. I.e., there is a distribution shift. There are mainly two kinds of OOD shifts: covariate shift and semantic shift. Covariate shift refers to a change in the input distribution. Semantic shift refers to a change in the output/label distribution. The authors of NAS-OOD  aim to develop a method that is robust to OOD data. They carry this out by jointly optimizing a conditional generator to synthesize OOD examples during training. To elaborate, let α, ω, θ G denote the parameters for the architecture topology, the supernet, and the conditional genetator G(·, ·), respectively. The conditional generator G(·, ·) takes data x and domain labels k as inputs. NAS-OOD is carried out by a minimax optimization. First, the conditional generator G is learned to generate novel domain data by maximizing the validation loss, serving as an adversarial attack. Next, the validation loss is minimized by optimizing the architecture variables in α on the generated OOD images. The constrained minimax optimization problem can be formulated as: where G(x, k) is the generated data from the original input data x on the domain labels k. The optimization is carried out by gradient ascent for θ G and gradient descent for ω and α. In addition, the authors apply gradient descent to θ G with respect to an auxiliary loss in order to improve the generator's consistency and semantic information preservation when training.

Poor Rank Correlation Alleviation
In this section, we will survey methods that aims to alleviate poor rank correlation. After supernet optimization, a subnet needs to be sampled and evaluated. This process repeats and returns a list of ranked architectures. Unfortunately, supernet-based NAS suffers from poor rank correlation. I.e., model accuracy predictions are poorly correlated to the standalone accuracy. [Bender et al., 2018] attributes this to the co-adaptation among the weights. I.e., when the shared weights have highly correlated behavior. Rank correlation is measured by three metrics: Kendall Tau (τ ) [Kendall, 1938], Spearman Rho (ρ), and Pearson R (R). Poor rank correlation alleviation is usually accomplished by navigating or factorizing the search space. Navigation is carried out by selecting a path [Bender et al., 2018;Chu et al., 2021;Chu et al., 2020]   The aforementioned methods fall under single path sampling, i.e., methods that sample only one architecture (subnet). An extension to this would be multi-path sampling, i.e., supernet optimization methods that sample multiple architectures. The authors of MixPath [Chu et al., 2020] find that a simple superposition of feature vectors sampled from multiple paths incurs very dynamic statistics causing training instability. They find that feature vectors from multiple paths are nearly multiples of those from a single-path. The solution to this is to scale the vectors down to the same magnitude creating stability. The authors accomplish said stability by proposing a novel Shadow Batch Normalization (SBN). MixPath is a two-stage pipeline: train supernet with SBN and searching for high performing models via NSGA-II.

Block-Wise Partitioning
It has been found that the rank correlation can vary depending on the search space itself [Zhang et al., 2020b]. For example, on the NAS-Bench-201 search space , the correlation can be as high as 0.7, whereas architectures sampled from DARTS-PTB  can perform worse than random search.
In an attempt to alleviate the co-adaptation, [Zhang et al., 2020a] proposed to reduce the weight sharing density (the number of architectures sharing the weight of one operator) by downscaling the search space. However, [Zhang et al., 2020b]  . The authors of Block-wisely Self-supervised Neural Architecture Search (BossNAS) achieve SOTA results alleviating the rank correlation issue by using block-wise weight-sharing.
We give a brief primer on block-wise partitioning. Let N be the supernet. We break N into N blocks by the supernet's depth and have: where N i+1 •N i denotes that the (i+1)-th block is connected to the i-th block in the supernet. Then each block is learned separately by optimizing: 2, . . . , N, (5) where L train , W i , A i , X, Y denote training loss, weights of the i-th block, search space of the i-th block, input data, and ground truth labels respectively. What inspired the authors of BossNAS to use block-wise weight-sharing were other works such as DNA  and DONNA [Moons et al., 2021] which achieve high correlation and high efficiency. However, the authors of Boss-NAS argue that those prior works have limited application on search spaces with disparate candidates, such as CNNs and Transformers . BossNAS trains each block separately before searching among all blocks in a linear combination of each block's validation loss: where α * is the optimal architecture, {α i } * is the same optimal architecture as a sequence of blocks, λ i is the weight of the i-th block, A is the search space, X i and Y i are the training data and labels for the i-th block respectively. On the MBConv Search Space, BossNAS produces strong rank correlations of τ = 0.65, ρ = 0.78, and R = 0.85. In contrast, DARTS , an early supernet-based method shows abysmal correlation τ = 0.08, ρ = 0.14 and R = 0.06.

Hierarchical Partitioning
The authors of few-shot NAS  adopt a hierarchical partitioning approach. They argue that one-shot NAS using only one supernet may not be able to model the search space due to its limited capacity and co-adaptation of operations [Bender et al., 2018]. The hierarchical partitioning is done by splitting the connections in the supernet. In a supernet, each connection is a compound edge. I.e., all predefined candidate operations connect a pair of nodes (feature maps). In one-shot NAS, only one subnet is sampled from the supernet. For the case of few-shot, subnets are collected hierarchically when splitting compound edges. After sampling k subnets, they are trained separately.
Naturally, there is a trade-off between evaluation accuracy and inference time. The hierarchical splitting could be repeated so that the search space is partitioned exhaustively (i.e. brute force search), but that would be computationally infeasible. The authors find that only using 7 subnets establishes new state-of-the-art results on ImageNet: 80.5% top-1 accuracy at 600 MB FLOPS and 77.5% top-1 accuracy at 238 MFLOPS.
The co-adaptation is found to be alleviated when measuring the rank correlation. The authors compared the correlation between the model's predicted performance and the ground truth. The Kendall's Tau scores for 1 sub-supernet (one-shot NAS), 6 sub-supernets, 36 sub-supernets, 216 subsupernets, 1296 sub-supernets (the entire search space) are 0.013, 0.12, 0.26, 0.63, and 1.0 respectively. Naturally, more accurate predictions are made when more of the search space is incorporated.

Supernet Training Consistency Shift
The search space itself is not the only avenue for finding poor rank correlation causes. The authors of  empirically demonstrate that supernet training consistency shift hurts architecture ranking correlation. They break the consistency shift into two components: feature shift and parameter shift.
Feature shift refers to network instability due to input image perturbation. Let X l , Y l , and W l denote the input, output, and weights of layer l respectively. Network instability refers to the disturbance in the loss L. By the chain rule for back propagation, we have: ∂L ∂W l = ∂L ∂Y l ∂Y l ∂W l = ∂L ∂Y l X l . This shows that architecture ranking preservation is highly dependent on the inputs X l . Since one-shot methods involve random pathsampling, the path that leads to layer l varies, and thus the input X l varies.
Parameter shift refers to contradictory parameter updates for a given layer. During supernet training, a given layer l is always present in different paths throughout all iterations. However, the weights might have a contradictory update when iterating. Looking at the gradient descent update there are two ways the rapidly varying W l will hurt rank correlation. One, the loss is dependent on both the weights W l and the descent W l − ∂L ∂W l . I.e., stable weights can ensure a correct loss descent and guarantee an accurate architecture ranking, while erratic parameters could not achieve a correct ranking. Two, since the input X l is generated by the network weights of the previous layers, varying parameters can also result in a feature shift, which further hurts architecture rank correlation.
The authors reduce the supernet training consistency shift by Π-NAS: a nontrivial supernet-Π model. They design a supernet-Π model and a nontrivial mean teacher model [Tarvainen and Valpola, 2017] to address feature shift and parameter shift respectively.

Transferable NAS
After supernet optimization, the sampled subnet requires post-processing: retraining, fine-tuning, etc. Training neural networks from scratch is computationally taxing, so there is interest in developing NAS methods that can yield architectures that can be instantly deployed for various tasks and hardware settings.

Specialized Networks for Specific Hardware Settings
Once-For-All (OFA) networks is a type of supernet that yields deployable sub-networks that require no further training for diverse tasks and hardware platforms [Cai et al., 2019]. The authors use progressive shrinking (PS) to train the OFA network. The idea is to initially train a large network and then progressively shrink it for fine-tuning with respect to 4 dimensions: width (channels), depth (layers), kernel size, and resolution. After training the full model, the authors set the filter size to be elastic, i.e., choose the kernel size to be 3, 5, or 7 at each layer while the depth and width remain the maximum values. The depth and width are then sequentially set to be elastic similarly. The resolution size remains elastic throughout the whole training process. For elastic kernel size, all kernels of different sizes share the same center. The concern that arises when kernels share the same center is that they need to play multiple roles (independent kernel and part of a large kernel) which can degrade performance. To address this, the authors use separate kernel transformation matrices for different layers. For elastic depth, if one wants a D layer unit from an N layer unit (such that D < N ), the authors keep the first D layers and skip the last N − D layers. For elastic width, the authors introduce a channel sorting operation to support partial widths. If full width is M , then the top K (K < M ) important channels are retained. Importance is defined as computing the L1 norm of a channel's weight (higher value meaning more important). The authors then use knowledge distillation [Hinton et al., 2015] after training the largest network.
After training the OFA network, the authors build neuralnetwork twins to predict the latency and accuracy given a neural network architecture. The authors randomly sample 16K sub-networks with different architectures and image resolutions, then measure their accuracy on 10K validation images. The goal is to build an accuracy predictor fed with architectures as an input signal and accuracy as labels. The latency for a given architecture is predicted by utilizing a latency lookup table based on various target hardware platforms. When given a target hardware and latency constraint, the authors conduct evolutionary search based on the neural-network-twins to yield a specialized sub-network.
Mobile setting performance is of great interest when it comes to OFA.

BigNAS
BigNAS [Yu et al., 2020] aims to train a single supernet that yields subnets whose sizes range from 200 to 1000 MFLOPS without extra training or post-processing. In other words, the weights can be directly deployed after training. The authors coin such supernet as a big single-stage model. They select architectures using a simple coarse-to-fine selection method to find the most accurate model under resource constraints such as FLOPs, memory footprint and/or runtime latency budgets on different devices. When training the single-stage model the authors use two techniques [Yu and Huang, 2019]: the sandwich rule and inplace distillation. The sandwich rule, in each training step, given a mini-batch of data, samples the smallest child model, the biggest (full) child model and N randomly sampled child models. It then aggregates the gradients from all sampled child models before updating the weights of the single-stage model. Here, "smallest" child means smallest input resolution, thinnest width, shallowest depth, and smallest kernel size. The motivation is to improve all child models in the search space simultaneously, by pushing up both the performance lower bound (smallest model) and the performance upper bound (full model).

Performance Comparison
Performance comparison can be done on multiple metrics: accuracy, latency, number of parameters, number of operations, search time, etc. On top of having a variety of different objectives, algorithms can be run on myriad environments and conditions: GPU models, number of GPUs, I/O overhead, etc. As a result, benchmarking NAS methods can be difficult. On top of this, NAS run on different search spaces can muddy the waters when benchmarking.
We provide a table listing different human and NAS designed architectures tasked to take on the ImageNet challenge. Supernet-based methods are highlighted and have their optimization strategy (poor rank correlation alleviation, transferability, data quality optimization) denoted. We also include top-1 accuracy, number of parameters, MACs, image resolution, and GPU days. Parameter and operation count are objective metrics that are only dependent on the algorithm. Image resolution is of concern because we are dealing with image classification; if a model performs well on a very low resolution, then it might be considered trivial. GPU days for search time would be heavily dependent on hardware specifications, but we include it because it has great practical importance. To offset high variance with respect to hardware, we include GPU settings for the models.

Future Directions
While this survey is about supernet optimization methods for image classification, the literature is lacking with regards to other computer vision tasks: semantic segmentation, object detection, etc. In other words, supernet research is primarily focused on achieving architectures that make the state-of-theart on ImageNet.
Most works are concerned with maximizing classification accuracy, reducing the search time, and making the sampled Table 1: Performance comparison among NAS methods on ImageNet. For optimization strategies, we abbreviated poor rank correlation alleviation as P, transferability as T, and data-side optimization as D. architectures as light-weight as possible. While reducing operation and parameter count can still see further improvement, classification accuracy is saturating. There is abundant NAS literature on model optimization for performance. In contrast, there is little literature on data-based optimization. This warrants a need for more works based on supernet optimization methods that are robust to OoD data. Future works can take inspiration from NAS-OoD  and modify the adversarial component to produce OoD data. Poor rank correlation is a ubiquitous issue in supernet optimization. Given the SOTA results of BossNAS  on ImageNet, future works can utilize the block-wise partitioning approach for the search space, which can lead to better performance and rank correlation alleviation.
State-of-the-art on ImageNet are neural networks utilizing Transformer with convolution and ViT models [Dai et al., 2021;Wortsman et al., 2022]. This clearly suggests that high performing networks can be built from Transformer-based architectures. This warrants building search spaces that include a Transformer component. Despite this precedent, there is little research done on Transformer-based search spaces in the NAS literature Ding et al., 2021].

Conclusion
Most NAS surveys focus on general NAS and puts less emphasis on supernet optimization. In addition, said surveys are structured by decomposing each method with respect to search space, search strategy, and performance estimation. This survey covers supernet optimization methods and adopts the structure from [Ren et al., 2020] by pairing challenges and solutions, and categorizing methods based on said pairs. This structure is used with the intention to build intuition on supernet NAS by thinking in terms of issues and respective fixes. We also provided a discussion on future directions with regards to research that is currently lacking in the supernet optimization literature.