Statistical Foundation of Variational Bayes Neural Networks
  Despite the popularism of Bayesian neural networks in recent years, its use
is somewhat limited in complex and big data situations due to the computational
cost associated with full posterior evaluations. Variational Bayes (VB)
provides a useful alternative to circumvent the computational cost and time
complexity associated with the generation of samples from the true posterior
using Markov Chain Monte Carlo (MCMC) techniques. The efficacy of the VB
methods is well established in machine learning literature. However, its
potential broader impact is hindered due to a lack of theoretical validity from
a statistical perspective. However there are few results which revolve around
the theoretical properties of VB, especially in non-parametric problems. In
this paper, we establish the fundamental result of posterior consistency for
the mean-field variational posterior (VP) for a feed-forward artificial neural
network model. The paper underlines the conditions needed to guarantee that the
VP concentrates around Hellinger neighborhoods of the true density function.
Additionally, the role of the scale parameter and its influence on the
convergence rates has also been discussed. The paper mainly relies on two
results (1) the rate at which the true posterior grows (2) the rate at which
the KL-distance between the posterior and variational posterior grows. The
theory provides a guideline of building prior distributions for Bayesian NN
models along with an assessment of accuracy of the corresponding VB
implementation.
