Information Theoretic Lower Bounds on Negative Log Likelihood
  In this article we use rate-distortion theory, a branch of information theory
devoted to the problem of lossy compression, to shed light on an important
problem in latent variable modeling of data: is there room to improve the
model? One way to address this question is to find an upper bound on the
probability (equivalently a lower bound on the negative log likelihood) that
the model can assign to some data as one varies the prior and/or the likelihood
function in a latent variable model. The core of our contribution is to
formally show that the problem of optimizing priors in latent variable models
is exactly an instance of the variational optimization problem that information
theorists solve when computing rate-distortion functions, and then to use this
to derive a lower bound on negative log likelihood. Moreover, we will show that
if changing the prior can improve the log likelihood, then there is a way to
change the likelihood function instead and attain the same log likelihood, and
thus rate-distortion theory is of relevance to both optimizing priors as well
as optimizing likelihood functions. We will experimentally argue for the
usefulness of quantities derived from rate-distortion theory in latent variable
modeling by applying them to a problem in image modeling.
