Analyzing and Mitigating Interference in Neural Architecture Search
  Weight sharing is a popular approach to reduce the cost of neural
architecture search (NAS) by reusing the weights of shared operators from
previously trained child models. However, the rank correlation between the
estimated accuracy and ground truth accuracy of those child models is low due
to the interference among different child models caused by weight sharing. In
this paper, we investigate the interference issue by sampling different child
models and calculating the gradient similarity of shared operators, and
observe: 1) the interference on a shared operator between two child models is
positively correlated with the number of different operators; 2) the
interference is smaller when the inputs and outputs of the shared operator are
more similar. Inspired by these two observations, we propose two approaches to
mitigate the interference: 1) MAGIC-T: rather than randomly sampling child
models for optimization, we propose a gradual modification scheme by modifying
one operator between adjacent optimization steps to minimize the interference
on the shared operators; 2) MAGIC-A: forcing the inputs and outputs of the
operator across all child models to be similar to reduce the interference.
Experiments on a BERT search space verify that mitigating interference via each
of our proposed methods improves the rank correlation of super-pet and
combining both methods can achieve better results. Our discovered architecture
outperforms RoBERTa$_{\rm base}$ by 1.1 and 0.6 points and ELECTRA$_{\rm base}$
by 1.6 and 1.1 points on the dev and test set of GLUE benchmark. Extensive
results on the BERT compression, reading comprehension and ImageNet task
demonstrate the effectiveness and generality of our proposed methods.
