Evaluating the Adversarial Robustness of Adaptive Test-time Defenses
  Adaptive defenses that use test-time optimization promise to improve
robustness to adversarial examples. We categorize such adaptive test-time
defenses and explain their potential benefits and drawbacks. In the process, we
evaluate some of the latest proposed adaptive defenses (most of them published
at peer-reviewed conferences). Unfortunately, none significantly improve upon
static models when evaluated appropriately. Some even weaken the underlying
static model while simultaneously increasing inference cost. While these
results are disappointing, we still believe that adaptive test-time defenses
are a promising avenue of research and, as such, we provide recommendations on
evaluating such defenses. We go beyond the checklist provided by Carlini et al.
(2019) by providing concrete steps that are specific to this type of defense.
