Parametric Adversarial Divergences are Good Losses for Generative
  Modeling
  Parametric adversarial divergences, which are a generalization of the losses
used to train generative adversarial networks (GANs), have often been described
as being approximations of their nonparametric counterparts, such as the
Jensen-Shannon divergence, which can be derived under the so-called optimal
discriminator assumption. In this position paper, we argue that despite being
"non-optimal", parametric divergences have distinct properties from their
nonparametric counterparts which can make them more suitable for learning
high-dimensional distributions. A key property is that parametric divergences
are only sensitive to certain aspects/moments of the distribution, which depend
on the architecture of the discriminator and the loss it was trained with. In
contrast, nonparametric divergences such as the Kullback-Leibler divergence are
sensitive to moments ignored by the discriminator, but they do not necessarily
correlate with sample quality (Theis et al., 2016). Similarly, we show that
mutual information can lead to unintuitive interpretations, and explore more
intuitive alternatives based on parametric divergences. We conclude that
parametric divergences are a flexible framework for defining statistical
quantities relevant to a specific modeling task.
