Measuring the False Sense of Security
  Recently, several papers have demonstrated how widespread gradient masking is
amongst proposed adversarial defenses. Defenses that rely on this phenomenon
are considered failed, and can easily be broken. Despite this, there has been
little investigation into ways of measuring the phenomenon of gradient masking
and enabling comparisons of its extent amongst different networks. In this
work, we investigate gradient masking under the lens of its mensurability,
departing from the idea that it is a binary phenomenon. We propose and motivate
several metrics for it, performing extensive empirical tests on defenses
suspected of exhibiting different degrees of gradient masking. These are
computationally cheaper than strong attacks, enable comparisons between models,
and do not require the large time investment of tailor-made attacks for
specific models. Our results reveal metrics that are successful in measuring
the extent of gradient masking across different networks
