Learning and Inference in Imaginary Noise Models
  Inspired by recent developments in learning smoothed densities with empirical
Bayes, we study variational autoencoders with a decoder that is tailored for
the random variable $Y=X+N(0,\sigma^2 I_d)$. A notion of smoothed variational
inference emerges where the smoothing is implicitly enforced by the noise model
of the decoder; "implicit", since during training the encoder only sees clean
samples. This is the concept of imaginary noise model, where the noise model
dictates the functional form of the variational lower bound
$\mathcal{L}(\sigma)$, but the noisy data are never seen during learning. The
model is named $\sigma$-VAE. We prove that all $\sigma$-VAEs are equivalent to
each other via a simple $\beta$-VAE expansion: $\mathcal{L}(\sigma_2) \equiv
\mathcal{L}(\sigma_1,\beta)$, where $\beta=\sigma_2^2/\sigma_1^2$. We prove a
similar result for the Laplace distribution in exponential families.
Empirically, we report an intriguing power law $\mathcal{D}_{\rm KL} \sim
\sigma^{-\nu}$ for the learned models and we study the inference in the
$\sigma$-VAE for unseen noisy data. The experiments were performed on MNIST,
where we show that quite remarkably the model can make reasonable inferences on
extremely noisy samples even though it has not seen any during training. The
vanilla VAE completely breaks down in this regime. We finish with a hypothesis
(the XYZ hypothesis) on the findings here.
