Making Method of Moments Great Again? -- How can GANs learn
  distributions
  Generative Adversarial Networks (GANs) are widely used models to learn
complex real-world distributions. In GANs, the training of the generator
usually stops when the discriminator can no longer distinguish the generator's
output from the set of training examples. A central question of GANs is that
when the training stops, whether the generated distribution is actually close
to the target distribution, and how the training process reaches to such
configurations efficiently? In this paper, we established a theoretical results
towards understanding this generator-discriminator training process. We
empirically observe that during the earlier stage of the GANs training, the
discriminator is trying to force the generator to match the low degree moments
between the generator's output and the target distribution. Moreover, only by
matching these empirical moments over polynomially many training examples, we
prove that the generator can already learn notable class of distributions,
including those that can be generated by two-layer neural networks.
