Spurious samples in deep generative models: bug or feature?
  Traditional wisdom in generative modeling literature is that spurious samples
that a model can generate are errors and they should be avoided. Recent
research, however, has shown interest in studying or even exploiting such
samples instead of eliminating them. In this paper, we ask the question whether
such samples can be eliminated all together without sacrificing coverage of the
generating distribution. For the class of models we consider, we experimentally
demonstrate that this is not possible without losing the ability to model some
of the test samples. While our results need to be confirmed on a broader set of
model families, these initial findings provide partial evidence that spurious
samples share structural properties with the learned dataset, which, in turn,
suggests they are not simply errors but a feature of deep generative nets.
