Weight-Sharing Neural Architecture Search: A Battle to Shrink the
  Optimization Gap
  Neural architecture search (NAS) has attracted increasing attentions in both
academia and industry. In the early age, researchers mostly applied individual
search methods which sample and evaluate the candidate architectures separately
and thus incur heavy computational overheads. To alleviate the burden,
weight-sharing methods were proposed in which exponentially many architectures
share weights in the same super-network, and the costly training procedure is
performed only once. These methods, though being much faster, often suffer the
issue of instability. This paper provides a literature review on NAS, in
particular the weight-sharing methods, and points out that the major challenge
comes from the optimization gap between the super-network and the
sub-architectures. From this perspective, we summarize existing approaches into
several categories according to their efforts in bridging the gap, and analyze
both advantages and disadvantages of these methodologies. Finally, we share our
opinions on the future directions of NAS and AutoML. Due to the expertise of
the authors, this paper mainly focuses on the application of NAS to computer
vision problems and may bias towards the work in our group.
