Fitness Landscape Footprint: A Framework to Compare Neural Architecture
  Search Problems
  Neural architecture search is a promising area of research dedicated to
automating the design of neural network models. This field is rapidly growing,
with a surge of methodologies ranging from Bayesian optimization,neuroevoltion,
to differentiable search, and applications in various contexts. However,
despite all great advances, few studies have presented insights on the
difficulty of the problem itself, thus the success (or fail) of these
methodologies remains unexplained. In this sense, the field of optimization has
developed methods that highlight key aspects to describe optimization problems.
The fitness landscape analysis stands out when it comes to characterize
reliably and quantitatively search algorithms. In this paper, we propose to use
fitness landscape analysis to study a neural architecture search problem.
Particularly, we introduce the fitness landscape footprint, an aggregation of
eight (8)general-purpose metrics to synthesize the landscape of an architecture
search problem. We studied two problems, the classical image classification
benchmark CIFAR-10, and the Remote-Sensing problem So2Sat LCZ42. The results
present a quantitative appraisal of the problems, allowing to characterize the
relative difficulty and other characteristics, such as the ruggedness or the
persistence, that helps to tailor a search strategy to the problem. Also, the
footprint is a tool that enables the comparison of multiple problems.
