On the Convergence of Gradient Descent in GANs: MMD GAN As a Gradient
  Flow
  We consider the maximum mean discrepancy ($\mathrm{MMD}$) GAN problem and
propose a parametric kernelized gradient flow that mimics the min-max game in
gradient regularized $\mathrm{MMD}$ GAN. We show that this flow provides a
descent direction minimizing the $\mathrm{MMD}$ on a statistical manifold of
probability distributions. We then derive an explicit condition which ensures
that gradient descent on the parameter space of the generator in gradient
regularized $\mathrm{MMD}$ GAN is globally convergent to the target
distribution. Under this condition, we give non asymptotic convergence results
of gradient descent in MMD GAN. Another contribution of this paper is the
introduction of a dynamic formulation of a regularization of $\mathrm{MMD}$ and
demonstrating that the parametric kernelized descent for $\mathrm{MMD}$ is the
gradient flow of this functional with respect to the new Riemannian structure.
Our obtained theoretical result allows ones to treat gradient flows for quite
general functionals and thus has potential applications to other types of
variational inferences on a statistical manifold beyond GANs. Finally,
numerical experiments suggest that our parametric kernelized gradient flow
stabilizes GAN training and guarantees convergence.
