How Does Supernet Help in Neural Architecture Search?
  Weight sharing, as an approach to speed up architecture performance
estimation has received wide attention. Instead of training each architecture
separately, weight sharing builds a supernet that assembles all the
architectures as its submodels. However, there has been debate over whether the
NAS process actually benefits from weight sharing, due to the gap between
supernet optimization and the objective of NAS. To further understand the
effect of weight sharing on NAS, we conduct a comprehensive analysis on five
search spaces, including NAS-Bench-101, NAS-Bench-201, DARTS-CIFAR10,
DARTS-PTB, and ProxylessNAS. We find that weight sharing works well on some
search spaces but fails on others. Taking a step forward, we further identified
biases accounting for such phenomenon and the capacity of weight sharing. Our
work is expected to inspire future NAS researchers to better leverage the power
of weight sharing.
