How to Train Your Super-Net: An Analysis of Training Heuristics in
  Weight-Sharing NAS
  Weight sharing promises to make neural architecture search (NAS) tractable
even on commodity hardware. Existing methods in this space rely on a diverse
set of heuristics to design and train the shared-weight backbone network,
a.k.a. the super-net. Since heuristics and hyperparameters substantially vary
across different methods, a fair comparison between them can only be achieved
by systematically analyzing the influence of these factors. In this paper, we
therefore provide a systematic evaluation of the heuristics and hyperparameters
that are frequently employed by weight-sharing NAS algorithms. Our analysis
uncovers that some commonly-used heuristics for super-net training negatively
impact the correlation between super-net and stand-alone performance, and
evidences the strong influence of certain hyperparameters and architectural
choices. Our code and experiments set a strong and reproducible baseline that
future works can build on.
