Neural Architecture Search on ImageNet in Four GPU Hours: A
  Theoretically Inspired Perspective
  Neural Architecture Search (NAS) has been explosively studied to automate the
discovery of top-performer neural networks. Current works require heavy
training of supernet or intensive architecture evaluations, thus suffering from
heavy resource consumption and often incurring search bias due to truncated
training or approximations. Can we select the best neural architectures without
involving any training and eliminate a drastic portion of the search cost? We
provide an affirmative answer, by proposing a novel framework called
training-free neural architecture search (TE-NAS). TE-NAS ranks architectures
by analyzing the spectrum of the neural tangent kernel (NTK) and the number of
linear regions in the input space. Both are motivated by recent theory advances
in deep networks and can be computed without any training and any label. We
show that: (1) these two measurements imply the trainability and expressivity
of a neural network; (2) they strongly correlate with the network's test
accuracy. Further on, we design a pruning-based NAS mechanism to achieve a more
flexible and superior trade-off between the trainability and expressivity
during the search. In NAS-Bench-201 and DARTS search spaces, TE-NAS completes
high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on
CIFAR-10 and ImageNet, respectively. We hope our work inspires more attempts in
bridging the theoretical findings of deep networks and practical impacts in
real NAS applications. Code is available at:
https://github.com/VITA-Group/TENAS.
