DEMI: Discriminative Estimator of Mutual Information
  Estimating mutual information between continuous random variables is often
intractable and extremely challenging for high-dimensional data. Recent
progress has leveraged neural networks to optimize variational lower bounds on
mutual information. Although showing promise for this difficult problem, the
variational methods have been theoretically and empirically proven to have
serious statistical limitations: 1) many methods struggle to produce accurate
estimates when the underlying mutual information is either low or high; 2) the
resulting estimators may suffer from high variance. Our approach is based on
training a classifier that provides the probability that a data sample pair is
drawn from the joint distribution rather than from the product of its marginal
distributions. Moreover, we establish a direct connection between mutual
information and the average log odds estimate produced by the classifier on a
test set, leading to a simple and accurate estimator of mutual information. We
show theoretically that our method and other variational approaches are
equivalent when they achieve their optimum, while our method sidesteps the
variational bound. Empirical results demonstrate high accuracy of our approach
and the advantages of our estimator in the context of representation learning.
Our demo is available at https://github.com/RayRuizhiLiao/demi_mi_estimator.
