Taking Care of The Discretization Problem: A Comprehensive Study of the
  Discretization Problem and A Black-Box Adversarial Attack in Discrete Integer
  Domain
  Numerous methods for crafting adversarial examples were proposed recently
with high success rate. Since most existing machine learning based classifiers
normalize images into some continuous, real vector, domain firstly, attacks
often craft adversarial examples in such domain. However, "adversarial"
examples may become benign after denormalizing them back into the discrete
integer domain, known as the discretization problem. This problem was mentioned
in some work, but has received relatively little attention.
  In this work, we first conduct a comprehensive study of existing methods and
tools for crafting. We theoretically analyze 34 representative methods and
empirically study 20 representative open source tools for crafting adversarial
images. Our study reveals that the discretization problem is far more serious
than originally thought. This suggests that the discretization problem should
be taken into account seriously when crafting adversarial examples and
measuring attack success rate. As a first step towards addressing this problem
in black-box scenario, we propose a black-box method which reduces the
adversarial example searching problem to a derivative-free optimization
problem. Our method is able to craft adversarial images by derivative-free
search in the discrete integer domain. Experimental results show that our
method is comparable to recent white-box methods (e.g., FGSM, BIM and C\&W) and
achieves significantly higher success rate in terms of adversarial examples in
the discrete integer domain than recent black-box methods (e.g., ZOO, NES-PGD
and Bandits). Moreover, our method is able to handle models that is
non-differentiable and successfully break the winner of NIPS 2017 competition
on defense with 95\% success rate. Our results suggest that discrete
optimization algorithms open up a promising area of research into effective
black-box attacks.
