Direct Adversarial Training: An Adaptive Method to Penalize Lipschitz
  Continuity of the Discriminator
  Generative Adversarial Networks (GANs) are the most popular image generation
models that have achieved remarkable performance on various tasks. However,
training instability is still one of the open problems for all GAN-based
algorithms. In order to stabilize GANs training, some regularization and
normalization techniques have been proposed to make the discriminator meet the
Lipschitz continuity. In this paper, a new approach inspired by works on
adversarial is proposed to stabilize the training process of GANs. It is found
that sometimes the images generated by the generator play a role just like
adversarial examples for discriminator during the training process, which might
be a part of the reason for the unstable training of GANs. With this discovery,
we propose a Direct Adversarial Training (DAT) method for the training process
of GANs to improve its performance. We prove that the DAT method can minimize
the Lipschitz constant of the discriminator adaptively. The advanced
performance of the proposed method is verified on multiple baseline and SOTA
networks, such as DCGAN, Spectral Normalization GAN, Self-supervised GAN, and
Information Maximum GAN. Code will be available at
\url{https://github.com/iceli1007/DAT-GAN}
