Differentiable Architecture Search Meets Network Pruning at
  Initialization: A More Reliable, Efficient, and Flexible Framework
  Although Differentiable ARchiTecture Search (DARTS) has become the mainstream
paradigm in Neural Architecture Search (NAS) due to its simplicity and
efficiency, more recent works found that the performance of the searched
architecture barely increases with the optimization proceeding in DARTS, and
the final magnitudes obtained by DARTS could hardly indicate the importance of
operations. The above observation reveal that the supervision signal in DARTS
may be a poor or unreliable indicator for the architecture search, inspiring an
interesting and promising direction: can we measure the operation importance
without any training under the differentiable paradigm? We provide an
affirmative answer by customizing the NAS as a network pruning at
initialization problem. With leveraging recently-proposed synaptic saliency
criteria in the network pruning at initialization, we seek to score the
importance of candidate operations in differentiable NAS without any training,
and proposed a novel framework called \textit{training free differentiable
architecture search} (FreeDARTS) accordingly. We show that, without any
training, FreeDARTS with different proxy metrics can outperform most NAS
baselines in different search spaces. More importantly, FreeDARTS is extremely
memory-efficient and computational-efficient as it abandons the training in the
architecture search phase, enabling FreeDARTS to perform architecture search on
a more flexible space and eliminate the depth gap between architecture search
and evaluation. We hope our work inspires more attempts in solving NAS from the
perspective of pruning at initialization.
