NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and
  Size
  Neural architecture search (NAS) has attracted a lot of attention and has
been illustrated to bring tangible benefits in a large number of applications
in the past few years. Architecture topology and architecture size have been
regarded as two of the most important aspects for the performance of deep
learning models and the community has spawned lots of searching algorithms for
both aspects of the neural architectures. However, the performance gain from
these searching algorithms is achieved under different search spaces and
training setups. This makes the overall performance of the algorithms to some
extent incomparable and the improvement from a sub-module of the searching
model unclear. In this paper, we propose NATS-Bench, a unified benchmark on
searching for both topology and size, for (almost) any up-to-date NAS
algorithm. NATS-Bench includes the search space of 15,625 neural cell
candidates for architecture topology and 32,768 for architecture size on three
datasets. We analyze the validity of our benchmark in terms of various criteria
and performance comparison of all candidates in the search space. We also show
the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS
algorithms on it. All logs and diagnostic information trained using the same
setup for each candidate are provided. This facilitates a much larger community
of researchers to focus on developing better NAS algorithms in a more
comparable and computationally cost friendly environment. All codes are
publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.
