Rethinking Uncertainty in Deep Learning: Whether and How it Improves
  Robustness
  Deep neural networks (DNNs) are known to be prone to adversarial attacks, for
which many remedies are proposed. While adversarial training (AT) is regarded
as the most robust defense, it suffers from poor performance both on clean
examples and under other types of attacks, e.g. attacks with larger
perturbations. Meanwhile, regularizers that encourage uncertain outputs, such
as entropy maximization (EntM) and label smoothing (LS) can maintain accuracy
on clean examples and improve performance under weak attacks, yet their ability
to defend against strong attacks is still in doubt. In this paper, we revisit
uncertainty promotion regularizers, including EntM and LS, in the field of
adversarial learning. We show that EntM and LS alone provide robustness only
under small perturbations. Contrarily, we show that uncertainty promotion
regularizers complement AT in a principled manner, consistently improving
performance on both clean examples and under various attacks, especially
attacks with large perturbations. We further analyze how uncertainty promotion
regularizers enhance the performance of AT from the perspective of Jacobian
matrices $\nabla_X f(X;\theta)$, and find out that EntM effectively shrinks the
norm of Jacobian matrices and hence promotes robustness.
