TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural
  Architecture Search
  Differentiable architecture search has gradually become the mainstream
research topic in the field of Neural Architecture Search (NAS) for its high
efficiency compared with the early NAS (EA-based, RL-based) methods. Recent
differentiable NAS also aims at further improving the search performance and
reducing the GPU-memory consumption. However, these methods are no longer
naturally capable of tackling the non-differentiable objectives, e.g., energy,
resource-constrained efficiency, and other metrics, let alone the
multi-objective search demands. Researches in the multi-objective NAS field
target this but requires vast computational resources cause of the sole
optimization of each candidate architecture. In light of this discrepancy, we
propose the TND-NAS, which is with the merits of the high efficiency in
differentiable NAS framework and the compatibility among non-differentiable
metrics in Multi-objective NAS. Under the differentiable NAS framework, with
the continuous relaxation of the search space, TND-NAS has the architecture
parameters ($\alpha$) been optimized in discrete space, while resorting to the
progressive search space shrinking by $\alpha$. Our representative experiment
takes two objectives (Parameters, Accuracy) as an example, we achieve a series
of high-performance compact architectures on CIFAR10 (1.09M/3.3\%, 2.4M/2.95\%,
9.57M/2.54\%) and CIFAR100 (2.46M/18.3\%, 5.46/16.73\%, 12.88/15.20\%)
datasets. Favorably, compared with other multi-objective NAS methods, TND-NAS
is less time-consuming (1.3 GPU-days on NVIDIA 1080Ti, 1/6 of that in
NSGA-Net), and can be conveniently adapted to real-world NAS scenarios
(resource-constrained, platform-specialized).
