Accelerated WGAN update strategy with loss change rate balancing
  Optimizing the discriminator in Generative Adversarial Networks (GANs) to
completion in the inner training loop is computationally prohibitive, and on
finite datasets would result in overfitting. To address this, a common update
strategy is to alternate between k optimization steps for the discriminator D
and one optimization step for the generator G. This strategy is repeated in
various GAN algorithms where k is selected empirically. In this paper, we show
that this update strategy is not optimal in terms of accuracy and convergence
speed, and propose a new update strategy for Wasserstein GANs (WGAN) and other
GANs using the WGAN loss(e.g. WGAN-GP, Deblur GAN, and Super-resolution GAN).
The proposed update strategy is based on a loss change ratio comparison of G
and D. We demonstrate that the proposed strategy improves both convergence
speed and accuracy.
