Generalizing Few-Shot NAS with Gradient Matching
  Efficient performance estimation of architectures drawn from large search
spaces is essential to Neural Architecture Search. One-Shot methods tackle this
challenge by training one supernet to approximate the performance of every
architecture in the search space via weight-sharing, thereby drastically
reducing the search cost. However, due to coupled optimization between child
architectures caused by weight-sharing, One-Shot supernet's performance
estimation could be inaccurate, leading to degraded search outcomes. To address
this issue, Few-Shot NAS reduces the level of weight-sharing by splitting the
One-Shot supernet into multiple separated sub-supernets via edge-wise
(layer-wise) exhaustive partitioning. Since each partition of the supernet is
not equally important, it necessitates the design of a more effective splitting
criterion. In this work, we propose a gradient matching score (GM) that
leverages gradient information at the shared weight for making informed
splitting decisions. Intuitively, gradients from different child models can be
used to identify whether they agree on how to update the shared modules, and
subsequently to decide if they should share the same weight. Compared with
exhaustive partitioning, the proposed criterion significantly reduces the
branching factor per edge. This allows us to split more edges (layers) for a
given budget, resulting in substantially improved performance as NAS search
spaces usually include dozens of edges (layers). Extensive empirical
evaluations of the proposed method on a wide range of search spaces
(NASBench-201, DARTS, MobileNet Space), datasets (cifar10, cifar100, ImageNet)
and search algorithms (DARTS, SNAS, RSPS, ProxylessNAS, OFA) demonstrate that
it significantly outperforms its Few-Shot counterparts while surpassing
previous comparable methods in terms of the accuracy of derived architectures.
