BaLeNAS: Differentiable Architecture Search via the Bayesian Learning
  Rule
  Differentiable Architecture Search (DARTS) has received massive attention in
recent years, mainly because it significantly reduces the computational cost
through weight sharing and continuous relaxation. However, more recent works
find that existing differentiable NAS techniques struggle to outperform naive
baselines, yielding deteriorative architectures as the search proceeds. Rather
than directly optimizing the architecture parameters, this paper formulates the
neural architecture search as a distribution learning problem through relaxing
the architecture weights into Gaussian distributions. By leveraging the
natural-gradient variational inference (NGVI), the architecture distribution
can be easily optimized based on existing codebases without incurring more
memory and computational consumption. We demonstrate how the differentiable NAS
benefits from Bayesian principles, enhancing exploration and improving
stability. The experimental results on NAS-Bench-201 and NAS-Bench-1shot1
benchmark datasets confirm the significant improvements the proposed framework
can make. In addition, instead of simply applying the argmax on the learned
parameters, we further leverage the recently-proposed training-free proxies in
NAS to select the optimal architecture from a group architectures drawn from
the optimized distribution, where we achieve state-of-the-art results on the
NAS-Bench-201 and NAS-Bench-1shot1 benchmarks. Our best architecture in the
DARTS search space also obtains competitive test errors with 2.37\%, 15.72\%,
and 24.2\% on CIFAR-10, CIFAR-100, and ImageNet datasets, respectively.
