An Adaptive View of Adversarial Robustness from Test-time Smoothing
  Defense
  The safety and robustness of learning-based decision-making systems are under
threats from adversarial examples, as imperceptible perturbations can mislead
neural networks to completely different outputs. In this paper, we present an
adaptive view of the issue via evaluating various test-time smoothing defense
against white-box untargeted adversarial examples. Through controlled
experiments with pretrained ResNet-152 on ImageNet, we first illustrate the
non-monotonic relation between adversarial attacks and smoothing defenses. Then
at the dataset level, we observe large variance among samples and show that it
is easy to inflate accuracy (even to 100%) or build large-scale (i.e., with
size ~10^4) subsets on which a designated method outperforms others by a large
margin. Finally at the sample level, as different adversarial examples require
different degrees of defense, the potential advantages of iterative methods are
also discussed. We hope this paper reveal useful behaviors of test-time
defenses, which could help improve the evaluation process for adversarial
robustness in the future.
