Noisy Differentiable Architecture Search
  Simplicity is the ultimate sophistication. Differentiable Architecture Search
(DARTS) has now become one of the mainstream paradigms of neural architecture
search. However, it largely suffers from the well-known performance collapse
issue due to the aggregation of skip connections. It is thought to have overly
benefited from the residual structure which accelerates the information flow.
To weaken this impact, we propose to inject unbiased random noise to impede the
flow. We name this novel approach NoisyDARTS. In effect, a network optimizer
should perceive this difficulty at each training step and refrain from
overshooting, especially on skip connections. In the long run, since we add no
bias to the gradient in terms of expectation, it is still likely to converge to
the right solution area. We also prove that the injected noise plays a role in
smoothing the loss landscape, which makes the optimization easier. Our method
features extreme simplicity and acts as a new strong baseline. We perform
extensive experiments across various search spaces, datasets, and tasks, where
we robustly achieve state-of-the-art results. Our code is available at
https://github.com/xiaomi-automl/NoisyDARTS.
