A Data-driven Approach to Neural Architecture Search Initialization
  Algorithmic design in neural architecture search (NAS) has received a lot of
attention, aiming to improve performance and reduce computational cost. Despite
the great advances made, few authors have proposed to tailor initialization
techniques for NAS. However, literature shows that a good initial set of
solutions facilitate finding the optima. Therefore, in this study, we propose a
data-driven technique to initialize a population-based NAS algorithm.
Particularly, we proposed a two-step methodology. First, we perform a
calibrated clustering analysis of the search space, and second, we extract the
centroids and use them to initialize a NAS algorithm. We benchmark our proposed
approach against random and Latin hypercube sampling initialization using three
population-based algorithms, namely a genetic algorithm, evolutionary
algorithm, and aging evolution, on CIFAR-10. More specifically, we use
NAS-Bench-101 to leverage the availability of NAS benchmarks. The results show
that compared to random and Latin hypercube sampling, the proposed
initialization technique enables achieving significant long-term improvements
for two of the search baselines, and sometimes in various search scenarios
(various training budgets). Moreover, we analyze the distributions of solutions
obtained and find that that the population provided by the data-driven
initialization technique enables retrieving local optima (maxima) of high
fitness and similar configurations.
