Sinkhorn Natural Gradient for Generative Models
  We consider the problem of minimizing a functional over a parametric family
of probability measures, where the parameterization is characterized via a
push-forward structure. An important application of this problem is in training
generative adversarial networks. In this regard, we propose a novel Sinkhorn
Natural Gradient (SiNG) algorithm which acts as a steepest descent method on
the probability space endowed with the Sinkhorn divergence. We show that the
Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit
expression and can be evaluated accurately in complexity that scales
logarithmically with respect to the desired accuracy. This is in sharp contrast
to existing natural gradient methods that can only be carried out
approximately. Moreover, in practical applications when only Monte-Carlo type
integration is available, we design an empirical estimator for SIM and provide
the stability analysis. In our experiments, we quantitatively compare SiNG with
state-of-the-art SGD-type solvers on generative tasks to demonstrate its
efficiency and efficacy of our method.
