Estimating Model Uncertainty of Neural Networks in Sparse Information
  Form
  We present a sparse representation of model uncertainty for Deep Neural
Networks (DNNs) where the parameter posterior is approximated with an inverse
formulation of the Multivariate Normal Distribution (MND), also known as the
information form. The key insight of our work is that the information matrix,
i.e. the inverse of the covariance matrix tends to be sparse in its spectrum.
Therefore, dimensionality reduction techniques such as low rank approximations
(LRA) can be effectively exploited. To achieve this, we develop a novel
sparsification algorithm and derive a cost-effective analytical sampler. As a
result, we show that the information form can be scalably applied to represent
model uncertainty in DNNs. Our exhaustive theoretical analysis and empirical
evaluations on various benchmarks show the competitiveness of our approach over
the current methods.
