Making targeted black-box evasion attacks effective and efficient
  We investigate how an adversary can optimally use its query budget for
targeted evasion attacks against deep neural networks in a black-box setting.
We formalize the problem setting and systematically evaluate what benefits the
adversary can gain by using substitute models. We show that there is an
exploration-exploitation tradeoff in that query efficiency comes at the cost of
effectiveness. We present two new attack strategies for using substitute models
and show that they are as effective as previous query-only techniques but
require significantly fewer queries, by up to three orders of magnitude. We
also show that an agile adversary capable of switching through different attack
techniques can achieve pareto-optimal efficiency. We demonstrate our attack
against Google Cloud Vision showing that the difficulty of black-box attacks
against real-world prediction APIs is significantly easier than previously
thought (requiring approximately 500 queries instead of approximately 20,000 as
in previous works).
