NAS evaluation is frustratingly hard
  Neural Architecture Search (NAS) is an exciting new field which promises to
be as much as a game-changer as Convolutional Neural Networks were in 2012.
Despite many great works leading to substantial improvements on a variety of
tasks, comparison between different methods is still very much an open issue.
While most algorithms are tested on the same datasets, there is no shared
experimental protocol followed by all. As such, and due to the under-use of
ablation studies, there is a lack of clarity regarding why certain methods are
more effective than others. Our first contribution is a benchmark of $8$ NAS
methods on $5$ datasets. To overcome the hurdle of comparing methods with
different search spaces, we propose using a method's relative improvement over
the randomly sampled average architecture, which effectively removes advantages
arising from expertly engineered search spaces or training protocols.
Surprisingly, we find that many NAS techniques struggle to significantly beat
the average architecture baseline. We perform further experiments with the
commonly used DARTS search space in order to understand the contribution of
each component in the NAS pipeline. These experiments highlight that: (i) the
use of tricks in the evaluation protocol has a predominant impact on the
reported performance of architectures; (ii) the cell-based search space has a
very narrow accuracy range, such that the seed has a considerable impact on
architecture rankings; (iii) the hand-designed macro-structure (cells) is more
important than the searched micro-structure (operations); and (iv) the
depth-gap is a real phenomenon, evidenced by the change in rankings between $8$
and $20$ cell architectures. To conclude, we suggest best practices, that we
hope will prove useful for the community and help mitigate current NAS
pitfalls. The code used is available at
https://github.com/antoyang/NAS-Benchmark.
