An Analysis of Super-Net Heuristics in Weight-Sharing NAS
  Weight sharing promises to make neural architecture search (NAS) tractable
even on commodity hardware. Existing methods in this space rely on a diverse
set of heuristics to design and train the shared-weight backbone network,
a.k.a. the super-net. Since heuristics substantially vary across different
methods and have not been carefully studied, it is unclear to which extent they
impact super-net training and hence the weight-sharing NAS algorithms. In this
paper, we disentangle super-net training from the search algorithm, isolate 14
frequently-used training heuristics, and evaluate them over three benchmark
search spaces. Our analysis uncovers that several commonly-used heuristics
negatively impact the correlation between super-net and stand-alone
performance, whereas simple, but often overlooked factors, such as proper
hyper-parameter settings, are key to achieve strong performance. Equipped with
this knowledge, we show that simple random search achieves competitive
performance to complex state-of-the-art NAS algorithms when the super-net is
properly trained.
