Variational Inference with Gaussian Mixture by Entropy Approximation
  Variational inference is a technique for approximating intractable posterior
distributions in order to quantify the uncertainty of machine learning.
Although the unimodal Gaussian distribution is usually chosen as a parametric
distribution, it hardly approximates the multimodality. In this paper, we
employ the Gaussian mixture distribution as a parametric distribution. A main
difficulty of variational inference with the Gaussian mixture is how to
approximate the entropy of the Gaussian mixture. We approximate the entropy of
the Gaussian mixture as the sum of the entropy of the unimodal Gaussian, which
can be analytically calculated. In addition, we theoretically analyze the
approximation error between the true entropy and approximated one in order to
reveal when our approximation works well. Specifically, the approximation error
is controlled by the ratios of the distances between the means to the sum of
the variances of the Gaussian mixture, and it converges to zero when the ratios
go to infinity. This situation seems to be more likely to occur in higher
dimensional weight spaces because of the curse of dimensionality. Therefore,
our result guarantees that our approximation works well, for example, in neural
networks that assume a large number of weights.
