Pi-NAS: Improving Neural Architecture Search by Reducing Supernet
  Training Consistency Shift
  Recently proposed neural architecture search (NAS) methods co-train billions
of architectures in a supernet and estimate their potential accuracy using the
network weights detached from the supernet. However, the ranking correlation
between the architectures' predicted accuracy and their actual capability is
incorrect, which causes the existing NAS methods' dilemma. We attribute this
ranking correlation problem to the supernet training consistency shift,
including feature shift and parameter shift. Feature shift is identified as
dynamic input distributions of a hidden layer due to random path sampling. The
input distribution dynamic affects the loss descent and finally affects
architecture ranking. Parameter shift is identified as contradictory parameter
updates for a shared layer lay in different paths in different training steps.
The rapidly-changing parameter could not preserve architecture ranking. We
address these two shifts simultaneously using a nontrivial supernet-Pi model,
called Pi-NAS. Specifically, we employ a supernet-Pi model that contains
cross-path learning to reduce the feature consistency shift between different
paths. Meanwhile, we adopt a novel nontrivial mean teacher containing negative
samples to overcome parameter shift and model collision. Furthermore, our
Pi-NAS runs in an unsupervised manner, which can search for more transferable
architectures. Extensive experiments on ImageNet and a wide range of downstream
tasks (e.g., COCO 2017, ADE20K, and Cityscapes) demonstrate the effectiveness
and universality of our Pi-NAS compared to supervised NAS. See Codes:
https://github.com/Ernie1/Pi-NAS.
