Single-DARTS: Towards Stable Architecture Search
  Differentiable architecture search (DARTS) marks a milestone in Neural
Architecture Search (NAS), boasting simplicity and small search costs. However,
DARTS still suffers from frequent performance collapse, which happens when some
operations, such as skip connections, zeroes and poolings, dominate the
architecture. In this paper, we are the first to point out that the phenomenon
is attributed to bi-level optimization. We propose Single-DARTS which merely
uses single-level optimization, updating network weights and architecture
parameters simultaneously with the same data batch. Even single-level
optimization has been previously attempted, no literature provides a systematic
explanation on this essential point. Replacing the bi-level optimization,
Single-DARTS obviously alleviates performance collapse as well as enhances the
stability of architecture search. Experiment results show that Single-DARTS
achieves state-of-the-art performance on mainstream search spaces. For
instance, on NAS-Benchmark-201, the searched architectures are nearly optimal
ones. We also validate that the single-level optimization framework is much
more stable than the bi-level one. We hope that this simple yet effective
method will give some insights on differential architecture search. The code is
available at https://github.com/PencilAndBike/Single-DARTS.git.
