Double Descent and Other Interpolation Phenomena in GANs
  We study overparameterization in generative adversarial networks (GANs) that
can interpolate the training data. We show that overparameterization can
improve generalization performance and accelerate the training process. We
study the generalization error as a function of latent space dimension and
identify two main behaviors, depending on the learning setting. First, we show
that overparameterized generative models that learn distributions by minimizing
a metric or $f$-divergence do not exhibit double descent in generalization
errors; specifically, all the interpolating solutions achieve the same
generalization error. Second, we develop a new pseudo-supervised learning
approach for GANs where the training utilizes pairs of fabricated (noise)
inputs in conjunction with real output samples. Our pseudo-supervised setting
exhibits double descent (and in some cases, triple descent) of generalization
errors. We combine pseudo-supervision with overparameterization (i.e., overly
large latent space dimension) to accelerate training while performing better,
or close to, the generalization performance without pseudo-supervision. While
our analysis focuses mostly on linear GANs, we also apply important insights
for improving generalization of nonlinear, multilayer GANs.
