Mutual Adversarial Training: Learning together is better than going
  alone
  Recent studies have shown that robustness to adversarial attacks can be
transferred across networks. In other words, we can make a weak model more
robust with the help of a strong teacher model. We ask if instead of learning
from a static teacher, can models "learn together" and "teach each other" to
achieve better robustness? In this paper, we study how interactions among
models affect robustness via knowledge distillation. We propose mutual
adversarial training (MAT), in which multiple models are trained together and
share the knowledge of adversarial examples to achieve improved robustness. MAT
allows robust models to explore a larger space of adversarial samples, and find
more robust feature spaces and decision boundaries. Through extensive
experiments on CIFAR-10 and CIFAR-100, we demonstrate that MAT can effectively
improve model robustness and outperform state-of-the-art methods under
white-box attacks, bringing $\sim$8% accuracy gain to vanilla adversarial
training (AT) under PGD-100 attacks. In addition, we show that MAT can also
mitigate the robustness trade-off among different perturbation types, bringing
as much as 13.1% accuracy gain to AT baselines against the union of $l_\infty$,
$l_2$ and $l_1$ attacks. These results show the superiority of the proposed
method and demonstrate that collaborative learning is an effective strategy for
designing robust models.
