Understanding and Accelerating Neural Architecture Search with
  Training-Free and Theory-Grounded Metrics
  This work targets designing a principled and unified training-free framework
for Neural Architecture Search (NAS), with high performance, low cost, and
in-depth interpretation. NAS has been explosively studied to automate the
discovery of top-performer neural networks, but suffers from heavy resource
consumption and often incurs search bias due to truncated training or
approximations. Recent NAS works start to explore indicators that can predict a
network's performance without training. However, they either leveraged limited
properties of deep networks, or the benefits of their training-free indicators
are not applied to more extensive search methods. By rigorous correlation
analysis, we present a unified framework to understand and accelerate NAS, by
disentangling "TEG" characteristics of searched networks - Trainability,
Expressivity, Generalization - all assessed in a training-free manner. The TEG
indicators could be scaled up and integrated with various NAS search methods,
including both supernet and single-path approaches. Extensive studies validate
the effective and efficient guidance from our TEG-NAS framework, leading to
both improved search accuracy and over 2.3x reduction in search time cost.
Moreover, we visualize search trajectories on three landscapes of "TEG"
characteristics, observing that while a good local minimum is easier to find on
NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is
much harder on the DARTS search space due to its complex landscape geometry.
Our code is available at https://github.com/VITA-Group/TEGNAS.
