ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and
  Gradients Accumulation
  Single-path based differentiable neural architecture search has great
strengths for its low computational cost and memory-friendly nature. However,
we surprisingly discover that it suffers from severe searching instability
which has been primarily ignored, posing a potential weakness for a wider
application. In this paper, we delve into its performance collapse issue and
propose a new algorithm called RObustifying Memory-Efficient NAS (ROME).
Specifically, 1) for consistent topology in the search and evaluation stage, we
involve separate parameters to disentangle the topology from the operations of
the architecture. In such a way, we can independently sample connections and
operations without interference; 2) to discount sampling unfairness and
variance, we enforce fair sampling for weight update and apply a gradient
accumulation mechanism for architecture parameters. Extensive experiments
demonstrate that our proposed method has strong performance and robustness,
where it mostly achieves state-of-the-art results on a large number of standard
benchmarks.
