NASI: Label- and Data-agnostic Neural Architecture Search at
  Initialization
  Recent years have witnessed a surging interest in Neural Architecture Search
(NAS). Various algorithms have been proposed to improve the search efficiency
and effectiveness of NAS, i.e., to reduce the search cost and improve the
generalization performance of the selected architectures, respectively.
However, the search efficiency of these algorithms is severely limited by the
need for model training during the search process. To overcome this limitation,
we propose a novel NAS algorithm called NAS at Initialization (NASI) that
exploits the capability of a Neural Tangent Kernel in being able to
characterize the converged performance of candidate architectures at
initialization, hence allowing model training to be completely avoided to boost
the search efficiency. Besides the improved search efficiency, NASI also
achieves competitive search effectiveness on various datasets like CIFAR-10/100
and ImageNet. Further, NASI is shown to be label- and data-agnostic under mild
conditions, which guarantees the transferability of architectures selected by
our NASI over different datasets.
