A critique of the DeepSec Platform for Security Analysis of Deep
  Learning Models
  At IEEE S&P 2019, the paper "DeepSec: A Uniform Platform for Security
Analysis of Deep Learning Model" aims to to "systematically evaluate the
existing adversarial attack and defense methods." While the paper's goals are
laudable, it fails to achieve them and presents results that are fundamentally
flawed and misleading. We explain the flaws in the DeepSec work, along with how
its analysis fails to meaningfully evaluate the various attacks and defenses.
Specifically, DeepSec (1) evaluates each defense obliviously, using attacks
crafted against undefended models; (2) evaluates attacks and defenses using
incorrect implementations that greatly under-estimate their effectiveness; (3)
evaluates the robustness of each defense as an average, not based on the most
effective attack against that defense; (4) performs several statistical
analyses incorrectly and fails to report variance; and, (5) as a result of
these errors draws invalid conclusions and makes sweeping generalizations.
