Unifying and Boosting Gradient-Based Training-Free Neural Architecture
  Search
  Neural architecture search (NAS) has gained immense popularity owing to its
ability to automate neural architecture design. A number of training-free
metrics are recently proposed to realize NAS without training, hence making NAS
more scalable. Despite their competitive empirical performances, a unified
theoretical understanding of these training-free metrics is lacking. As a
consequence, (a) the relationships among these metrics are unclear, (b) there
is no theoretical guarantee for their empirical performances and
transferability, and (c) there may exist untapped potential in training-free
NAS, which can be unveiled through a unified theoretical understanding. To this
end, this paper presents a unified theoretical analysis of gradient-based
training-free NAS, which allows us to (a) theoretically study their
relationships, (b) theoretically guarantee their generalization performances
and transferability, and (c) exploit our unified theoretical understanding to
develop a novel framework named hybrid NAS (HNAS) which consistently boosts
training-free NAS in a principled way. Interestingly, HNAS is able to enjoy the
advantages of both training-free (i.e., superior search efficiency) and
training-based (i.e., remarkable search effectiveness) NAS, which we have
demonstrated through extensive experiments.
