On Success and Simplicity: A Second Look at Transferable Targeted
  Attacks
  Achieving transferability of targeted attacks is reputed to be remarkably
difficult. Currently, state-of-the-art approaches are resource-intensive
because they necessitate training model(s) for each target class with
additional data. In our investigation, we find, however, that simple
transferable attacks which require neither additional data nor model training
can achieve surprisingly high targeted transferability. This insight has been
overlooked until now, mainly due to the widespread practice of unreasonably
restricting attack optimization to a limited number of iterations. In
particular, we, for the first time, identify that a simple logit loss can yield
competitive results with the state of the arts. Our analysis spans a variety of
transfer settings, especially including three new, realistic settings: an
ensemble transfer setting with little model similarity, a worse-case setting
with low-ranked target classes, and also a real-world attack against the Google
Cloud Vision API. Results in these new settings demonstrate that the commonly
adopted, easy settings cannot fully reveal the actual properties of different
attacks and may cause misleading comparisons. We also show the usefulness of
the simple logit loss for generating targeted universal adversarial
perturbations in a data-free and training-free manner. Overall, the aim of our
analysis is to inspire a more meaningful evaluation on targeted
transferability. Code is available at
https://github.com/ZhengyuZhao/Targeted-Tansfer
