$\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture
  Search
  Neural Architecture Search~(NAS) has attracted increasingly more attention in
recent years because of its capability to design deep neural networks
automatically. Among them, differential NAS approaches such as DARTS, have
gained popularity for the search efficiency. However, they suffer from two main
issues, the weak robustness to the performance collapse and the poor
generalization ability of the searched architectures. To solve these two
problems, a simple-but-efficient regularization method, termed as Beta-Decay,
is proposed to regularize the DARTS-based NAS searching process. Specifically,
Beta-Decay regularization can impose constraints to keep the value and variance
of activated architecture parameters from too large. Furthermore, we provide
in-depth theoretical analysis on how it works and why it works. Experimental
results on NAS-Bench-201 show that our proposed method can help to stabilize
the searching process and makes the searched network more transferable across
different datasets. In addition, our search scheme shows an outstanding
property of being less dependent on training time and data. Comprehensive
experiments on a variety of search spaces and datasets validate the
effectiveness of the proposed method.
