Deeper Insights into Weight Sharing in Neural Architecture Search
  With the success of deep neural networks, Neural Architecture Search (NAS) as
a way of automatic model design has attracted wide attention. As training every
child model from scratch is very time-consuming, recent works leverage
weight-sharing to speed up the model evaluation procedure. These approaches
greatly reduce computation by maintaining a single copy of weights on the
super-net and share the weights among every child model. However,
weight-sharing has no theoretical guarantee and its impact has not been well
studied before. In this paper, we conduct comprehensive experiments to reveal
the impact of weight-sharing: (1) The best-performing models from different
runs or even from consecutive epochs within the same run have significant
variance; (2) Even with high variance, we can extract valuable information from
training the super-net with shared weights; (3) The interference between child
models is a main factor that induces high variance; (4) Properly reducing the
degree of weight sharing could effectively reduce variance and improve
performance.
