Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
  With the rapid development of neural architecture search (NAS), researchers
found powerful network architectures for a wide range of vision tasks. However,
it remains unclear if the searched architecture can transfer across different
types of tasks as manually designed ones did. This paper puts forward this
problem, referred to as NAS in the wild, which explores the possibility of
finding the optimal architecture in a proxy dataset and then deploying it to
mostly unseen scenarios.
  We instantiate this setting using a currently popular algorithm named
differentiable architecture search (DARTS), which often suffers unsatisfying
performance while being transferred across different tasks. We argue that the
accuracy drop originates from the formulation that uses a super-network for
search but a sub-network for re-training. The different properties of these
stages have resulted in a significant optimization gap, and consequently, the
architectural parameters "over-fit" the super-network. To alleviate the gap, we
present a progressive method that gradually increases the network depth during
the search stage, which leads to the Progressive DARTS (P-DARTS) algorithm.
With a reduced search cost (7 hours on a single GPU), P-DARTS achieves improved
performance on both the proxy dataset (CIFAR10) and a few target problems
(ImageNet classification, COCO detection and three ReID benchmarks). Our code
is available at \url{https://github.com/chenxin061/pdarts}.
