A Study of the Learning Progress in Neural Architecture Search
  Techniques
  In neural architecture search, the structure of the neural network to best
model a given dataset is determined by an automated search process. Efficient
Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently
received considerable attention due to its ability to find excellent
architectures within a comparably short search time. In this work, which is
motivated by the quest to further improve the learning speed of architecture
search, we evaluate the learning progress of the controller which generates the
architectures in ENAS. We measure the progress by comparing the architectures
generated by it at different controller training epochs, where architectures
are evaluated after having re-trained them from scratch. As a surprising
result, we find that the learning curves are completely flat, i.e., there is no
observable progress of the controller in terms of the performance of its
generated architectures. This observation is consistent across the CIFAR-10 and
CIFAR-100 datasets and two different search spaces. We conclude that the high
quality of the models generated by ENAS is a result of the search space design
rather than the controller training, and our results indicate that one-shot
architecture design is an efficient alternative to architecture search by ENAS.
