Multi-fidelity Neural Architecture Search with Knowledge Distillation
  Neural architecture search (NAS) targets at finding the optimal architecture
of a neural network for a problem or a family of problems. Evaluations of
neural architectures are very time-consuming. One of the possible ways to
mitigate this issue is to use low-fidelity evaluations, namely training on a
part of a dataset, fewer epochs, with fewer channels, etc. In this paper, we
propose a bayesian multi-fidelity method for neural architecture search: MF-KD.
The method relies on a new approach to low-fidelity evaluations of neural
architectures by training for a few epochs using a knowledge distillation.
Knowledge distillation adds to a loss function a term forcing a network to
mimic some teacher network. We carry out experiments on CIFAR-10, CIFAR-100,
and ImageNet-16-120. We show that training for a few epochs with such a
modified loss function leads to a better selection of neural architectures than
training for a few epochs with a logistic loss. The proposed method outperforms
several state-of-the-art baselines.
