Generalized Denoising Auto-Encoders as Generative Models
  Recent work has shown how denoising and contractive autoencoders implicitly
capture the structure of the data-generating density, in the case where the
corruption noise is Gaussian, the reconstruction error is the squared error,
and the data is continuous-valued. This has led to various proposals for
sampling from this implicitly learned density function, using Langevin and
Metropolis-Hastings MCMC. However, it remained unclear how to connect the
training procedure of regularized auto-encoders to the implicit estimation of
the underlying data-generating distribution when the data are discrete, or
using other forms of corruption process and reconstruction errors. Another
issue is the mathematical justification which is only valid in the limit of
small corruption noise. We propose here a different attack on the problem,
which deals with all these issues: arbitrary (but noisy enough) corruption,
arbitrary reconstruction loss (seen as a log-likelihood), handling both
discrete and continuous-valued variables, and removing the bias due to
non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).
