DARTS-PRIME: Regularization and Scheduling Improve Constrained
  Optimization in Differentiable NAS
  Differentiable Architecture Search (DARTS) is a recent neural architecture
search (NAS) method based on a differentiable relaxation. Due to its success,
numerous variants analyzing and improving parts of the DARTS framework have
recently been proposed. By considering the problem as a constrained bilevel
optimization, we present and analyze DARTS-PRIME, a variant including
improvements to architectural weight update scheduling and regularization
towards discretization. We propose a dynamic schedule based on per-minibatch
network information to make architecture updates more informed, as well as
proximity regularization to promote well-separated discretization. Our results
in multiple domains show that DARTS-PRIME improves both performance and
reliability, comparable to state-of-the-art in differentiable NAS.
