Batch Normalization is a Cause of Adversarial Vulnerability
  Batch normalization (batch norm) is often used in an attempt to stabilize and
accelerate training in deep neural networks. In many cases it indeed decreases
the number of parameter updates required to achieve low training error.
However, it also reduces robustness to small adversarial input perturbations
and noise by double-digit percentages, as we show on five standard datasets.
Furthermore, substituting weight decay for batch norm is sufficient to nullify
the relationship between adversarial vulnerability and the input dimension. Our
work is consistent with a mean-field analysis that found that batch norm causes
exploding gradients.
