Information Theory with Kernel Methods
  We consider the analysis of probability distributions through their
associated covariance operators from reproducing kernel Hilbert spaces. We show
that the von Neumann entropy and relative entropy of these operators are
intimately related to the usual notions of Shannon entropy and relative
entropy, and share many of their properties. They come together with efficient
estimation algorithms from various oracles on the probability distributions. We
also consider product spaces and show that for tensor product kernels, we can
define notions of mutual information and joint entropies, which can then
characterize independence perfectly, but only partially conditional
independence. We finally show how these new notions of relative entropy lead to
new upper-bounds on log partition functions, that can be used together with
convex optimization within variational inference methods, providing a new
family of probabilistic inference methods.
