$α$NAS: Neural Architecture Search using Property Guided Synthesis
In the past few years, neural architecture search (NAS) has become an
increasingly important tool within the deep learning community. Despite the
many recent successes of NAS, however, most existing approaches operate within
highly structured design spaces, and hence explore only a small fraction of the
full search space of neural architectures while also requiring significant
manual effort from domain experts. In this work, we develop techniques that
enable efficient NAS in a significantly larger design space. To accomplish
this, we propose to perform NAS in an abstract search space of program
properties. Our key insights are as follows: (1) the abstract search space is
significantly smaller than the original search space, and (2) architectures
with similar program properties also have similar performance; thus, we can
search more efficiently in the abstract search space. To enable this approach,
we also propose a novel efficient synthesis procedure, which accepts a set of
promising program properties, and returns a satisfying neural architecture. We
implement our approach, $\alpha$NAS, within an evolutionary framework, where
the mutations are guided by the program properties. Starting with a ResNet-34
model, $\alpha$NAS produces a model with slightly improved accuracy on CIFAR-10
but 96% fewer parameters. On ImageNet, $\alpha$NAS is able to improve over
Vision Transformer (30% fewer FLOPS and parameters), ResNet-50 (23% fewer
FLOPS, 14% fewer parameters), and EfficientNet (7% fewer FLOPS and parameters)
without any degradation in accuracy.