FEAR: A Simple Lightweight Method to Rank Architectures
  The fundamental problem in Neural Architecture Search (NAS) is to efficiently
find high-performing architectures from a given search space. We propose a
simple but powerful method which we call FEAR, for ranking architectures in any
search space. FEAR leverages the viewpoint that neural networks are powerful
non-linear feature extractors. First, we train different architectures in the
search space to the same training or validation error. Then, we compare the
usefulness of the features extracted by each architecture. We do so with a
quick training keeping most of the architecture frozen. This gives fast
estimates of the relative performance. We validate FEAR on Natsbench topology
search space on three different datasets against competing baselines and show
strong ranking correlation especially compared to recently proposed zero-cost
methods. FEAR particularly excels at ranking high-performance architectures in
the search space. When used in the inner loop of discrete search algorithms
like random search, FEAR can cut down the search time by approximately 2.4X
without losing accuracy. We additionally empirically study very recently
proposed zero-cost measures for ranking and find that they breakdown in ranking
performance as training proceeds and also that data-agnostic ranking scores
which ignore the dataset do not generalize across dissimilar datasets.
