Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture
  Search
  Differentiable Architecture Search (DARTS) is now a widely disseminated
weight-sharing neural architecture search method. However, it suffers from
well-known performance collapse due to an inevitable aggregation of skip
connections. In this paper, we first disclose that its root cause lies in an
unfair advantage in exclusive competition. Through experiments, we show that if
either of two conditions is broken, the collapse disappears. Thereby, we
present a novel approach called Fair DARTS where the exclusive competition is
relaxed to be collaborative. Specifically, we let each operation's
architectural weight be independent of others. Yet there is still an important
issue of discretization discrepancy. We then propose a zero-one loss to push
architectural weights towards zero or one, which approximates an expected
multi-hot solution. Our experiments are performed on two mainstream search
spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet.
Our code is available on https://github.com/xiaomi-automl/fairdarts .
