Zero-Cost Proxies Meet Differentiable Architecture Search
  Differentiable neural architecture search (NAS) has attracted significant
attention in recent years due to its ability to quickly discover promising
architectures of deep neural networks even in very large search spaces. Despite
its success, DARTS lacks robustness in certain cases, e.g. it may degenerate to
trivial architectures with excessive parametric-free operations such as skip
connection or random noise, leading to inferior performance. In particular,
operation selection based on the magnitude of architectural parameters was
recently proven to be fundamentally wrong showcasing the need to rethink this
aspect. On the other hand, zero-cost proxies have been recently studied in the
context of sample-based NAS showing promising results -- speeding up the search
process drastically in some cases but also failing on some of the large search
spaces typical for differentiable NAS. In this work we propose a novel
operation selection paradigm in the context of differentiable NAS which
utilises zero-cost proxies. Our perturbation-based zero-cost operation
selection (Zero-Cost-PT) improves searching time and, in many cases, accuracy
compared to the best available differentiable architecture search, regardless
of the search space size. Specifically, we are able to find comparable
architectures to DARTS-PT on the DARTS CNN search space while being over 40x
faster (total searching time 25 minutes on a single GPU).
