Eigenvalue Corrected Noisy Natural Gradient
  Variational Bayesian neural networks combine the flexibility of deep learning
with Bayesian uncertainty estimation. However, inference procedures for
flexible variational posteriors are computationally expensive. A recently
proposed method, noisy natural gradient, is a surprisingly simple method to fit
expressive posteriors by adding weight noise to regular natural gradient
updates. Noisy K-FAC is an instance of noisy natural gradient that fits a
matrix-variate Gaussian posterior with minor changes to ordinary K-FAC.
Nevertheless, a matrix-variate Gaussian posterior does not capture an accurate
diagonal variance. In this work, we extend on noisy K-FAC to obtain a more
flexible posterior distribution called eigenvalue corrected matrix-variate
Gaussian. The proposed method computes the full diagonal re-scaling factor in
Kronecker-factored eigenbasis. Empirically, our approach consistently
outperforms existing algorithms (e.g., noisy K-FAC) on regression and
classification tasks.
