On Redundancy and Diversity in Cell-based Neural Architecture Search
  Searching for the architecture cells is a dominant paradigm in NAS. However,
little attention has been devoted to the analysis of the cell-based search
spaces even though it is highly important for the continual development of NAS.
In this work, we conduct an empirical post-hoc analysis of architectures from
the popular cell-based search spaces and find that the existing search spaces
contain a high degree of redundancy: the architecture performance is minimally
sensitive to changes at large parts of the cells, and universally adopted
designs, like the explicit search for a reduction cell, significantly increase
the complexities but have very limited impact on the performance. Across
architectures found by a diverse set of search strategies, we consistently find
that the parts of the cells that do matter for architecture performance often
follow similar and simple patterns. By explicitly constraining cells to include
these patterns, randomly sampled architectures can match or even outperform the
state of the art. These findings cast doubts into our ability to discover truly
novel architectures in the existing cell-based search spaces, and inspire our
suggestions for improvement to guide future NAS research. Code is available at
https://github.com/xingchenwan/cell-based-NAS-analysis.
