Hierarchical Neural Architecture Search via Operator Clustering
  Recently, the efficiency of automatic neural architecture design has been
significantly improved by gradient-based search methods such as DARTS. However,
recent literature has brought doubt to the generalization ability of DARTS,
arguing that DARTS performs poorly when the search space is changed, i.e, when
different set of candidate operators are used. Regularization techniques such
as early stopping have been proposed to partially solve this problem. In this
paper, we tackle this problem from a different perspective by identifying two
contributing factors to the collapse of DARTS when the search space changes:
(1) the correlation of similar operators incurs unfavorable competition among
them and makes their relative importance score unreliable and (2) the
optimization complexity gap between the proxy search stage and the final
training. Based on these findings, we propose a new hierarchical search
algorithm. With its operator clustering and optimization complexity match, the
algorithm can consistently find high-performance architecture across various
search spaces. For all the five variants of the popular cell-based search
spaces, the proposed algorithm always obtains state-of-the-art architecture
with best accuracy on the CIFAR-10, CIFAR-100 and ImageNet over other
well-established DARTS-alike algorithms. Code is available at
https://github.com/susan0199/StacNAS.
