EcoNAS: Finding Proxies for Economical Neural Architecture Search
  Neural Architecture Search (NAS) achieves significant progress in many
computer vision tasks. While many methods have been proposed to improve the
efficiency of NAS, the search progress is still laborious because training and
evaluating plausible architectures over large search space is time-consuming.
Assessing network candidates under a proxy (i.e., computationally reduced
setting) thus becomes inevitable. In this paper, we observe that most existing
proxies exhibit different behaviors in maintaining the rank consistency among
network candidates. In particular, some proxies can be more reliable -- the
rank of candidates does not differ much comparing their reduced setting
performance and final performance. In this paper, we systematically investigate
some widely adopted reduction factors and report our observations. Inspired by
these observations, we present a reliable proxy and further formulate a
hierarchical proxy strategy. The strategy spends more computations on candidate
networks that are potentially more accurate, while discards unpromising ones in
early stage with a fast proxy. This leads to an economical evolutionary-based
NAS (EcoNAS), which achieves an impressive 400x search time reduction in
comparison to the evolutionary-based state of the art (8 vs. 3150 GPU days).
Some new proxies led by our observations can also be applied to accelerate
other NAS methods while still able to discover good candidate networks with
performance matching those found by previous proxy strategies.
