Detecting Overfitting of Deep Generative Networks via Latent Recovery
  State of the art deep generative networks are capable of producing images
with such incredible realism that they can be suspected of memorizing training
images. It is why it is not uncommon to include visualizations of training set
nearest neighbors, to suggest generated images are not simply memorized. We
demonstrate this is not sufficient and motivates the need to study
memorization/overfitting of deep generators with more scrutiny. This paper
addresses this question by i) showing how simple losses are highly effective at
reconstructing images for deep generators ii) analyzing the statistics of
reconstruction errors when reconstructing training and validation images, which
is the standard way to analyze overfitting in machine learning. Using this
methodology, this paper shows that overfitting is not detectable in the pure
GAN models proposed in the literature, in contrast with those using hybrid
adversarial losses, which are amongst the most widely applied generative
methods. The paper also shows that standard GAN evaluation metrics fail to
capture memorization for some deep generators. Finally, the paper also shows
how off-the-shelf GAN generators can be successfully applied to face inpainting
and face super-resolution using the proposed reconstruction method, without
hybrid adversarial losses.
