The Variational Predictive Natural Gradient
  Variational inference transforms posterior inference into parametric
optimization thereby enabling the use of latent variable models where otherwise
impractical. However, variational inference can be finicky when different
variational parameters control variables that are strongly correlated under the
model. Traditional natural gradients based on the variational approximation
fail to correct for correlations when the approximation is not the true
posterior. To address this, we construct a new natural gradient called the
Variational Predictive Natural Gradient (VPNG). Unlike traditional natural
gradients for variational inference, this natural gradient accounts for the
relationship between model parameters and variational parameters. We
demonstrate the insight with a simple example as well as the empirical value on
a classification task, a deep generative model of images, and probabilistic
matrix factorization for recommendation.
