Enhancing Adversarial Training with Second-Order Statistics of Weights
  Adversarial training has been shown to be one of the most effective
approaches to improve the robustness of deep neural networks. It is formalized
as a min-max optimization over model weights and adversarial perturbations,
where the weights can be optimized through gradient descent methods like SGD.
In this paper, we show that treating model weights as random variables allows
for enhancing adversarial training through \textbf{S}econd-Order
\textbf{S}tatistics \textbf{O}ptimization (S$^2$O) with respect to the weights.
By relaxing a common (but unrealistic) assumption of previous PAC-Bayesian
frameworks that all weights are statistically independent, we derive an
improved PAC-Bayesian adversarial generalization bound, which suggests that
optimizing second-order statistics of weights can effectively tighten the
bound. In addition to this theoretical insight, we conduct an extensive set of
experiments, which show that S$^2$O not only improves the robustness and
generalization of the trained neural networks when used in isolation, but also
integrates easily in state-of-the-art adversarial training techniques like
TRADES, AWP, MART, and AVMixup, leading to a measurable improvement of these
techniques. The code is available at \url{https://github.com/Alexkael/S2O}.
