Sample weighting as an explanation for mode collapse in generative
  adversarial networks
  Generative adversarial networks were introduced with a logistic MiniMax cost
formulation, which normally fails to train due to saturation, and a
Non-Saturating reformulation. While addressing the saturation problem, NS-GAN
also inverts the generator's sample weighting, implicitly shifting emphasis
from higher-scoring to lower-scoring samples when updating parameters. We
present both theory and empirical results suggesting that this makes NS-GAN
prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample
weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient
such that its magnitude approximates NS-GAN's gradient magnitude. MM-nsat has
qualitatively different training dynamics, and on MNIST and CIFAR-10 it is
stronger in terms of mode coverage, stability and FID. While the empirical
results for MM-nsat are promising and favorable also in comparison with the
LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why
NS-GAN's sample weighting causes mode dropping and training collapse.
