Searching for Robust Neural Architectures via Comprehensive and Reliable
  Evaluation
  Neural architecture search (NAS) could help search for robust network
architectures, where defining robustness evaluation metrics is the important
procedure. However, current robustness evaluations in NAS are not sufficiently
comprehensive and reliable. In particular, the common practice only considers
adversarial noise and quantified metrics such as the Jacobian matrix, whereas,
some studies indicated that the models are also vulnerable to other types of
noises such as natural noise. In addition, existing methods taking adversarial
noise as the evaluation just use the robust accuracy of the FGSM or PGD, but
these adversarial attacks could not provide the adequately reliable evaluation,
leading to the vulnerability of the models under stronger attacks. To alleviate
the above problems, we propose a novel framework, called Auto Adversarial
Attack and Defense (AAAD), where we employ neural architecture search methods,
and four types of robustness evaluations are considered, including adversarial
noise, natural noise, system noise and quantified metrics, thereby assisting in
finding more robust architectures. Also, among the adversarial noise, we use
the composite adversarial attack obtained by random search as the new metric to
evaluate the robustness of the model architectures. The empirical results on
the CIFAR10 dataset show that the searched efficient attack could help find
more robust architectures.
