Semi-Implicit Hybrid Gradient Methods with Application to Adversarial
  Robustness
  Adversarial examples, crafted by adding imperceptible perturbations to
natural inputs, can easily fool deep neural networks (DNNs). One of the most
successful methods for training adversarially robust DNNs is solving a
nonconvex-nonconcave minimax problem with an adversarial training (AT)
algorithm. However, among the many AT algorithms, only Dynamic AT (DAT) and You
Only Propagate Once (YOPO) guarantee convergence to a stationary point. In this
work, we generalize the stochastic primal-dual hybrid gradient algorithm to
develop semi-implicit hybrid gradient methods (SI-HGs) for finding stationary
points of nonconvex-nonconcave minimax problems. SI-HGs have the convergence
rate $O(1/K)$, which improves upon the rate $O(1/K^{1/2})$ of DAT and YOPO. We
devise a practical variant of SI-HGs, and show that it outperforms other AT
algorithms in terms of convergence speed and robustness.
