Analytic Mutual Information in Bayesian Neural Networks
  Bayesian neural networks have successfully designed and optimized a robust
neural network model in many application problems, including uncertainty
quantification. However, with its recent success, information-theoretic
understanding about the Bayesian neural network is still at an early stage.
Mutual information is an example of an uncertainty measure in a Bayesian neural
network to quantify epistemic uncertainty. Still, no analytic formula is known
to describe it, one of the fundamental information measures to understand the
Bayesian deep learning framework. In this paper, we derive the analytical
formula of the mutual information between model parameters and the predictive
output by leveraging the notion of the point process entropy. Then, as an
application, we discuss the parameter estimation of the Dirichlet distribution
and show its practical application in the active learning uncertainty measures
by demonstrating that our analytical formula can improve the performance of
active learning further in practice.
