Estimating Kullback-Leibler Divergence Using Kernel Machines
  Recently, a method called the Mutual Information Neural Estimator (MINE) that
uses neural networks has been proposed to estimate mutual information and more
generally the Kullback-Leibler (KL) divergence between two distributions. The
method uses the Donsker-Varadhan representation to arrive at the estimate of
the KL divergence and is better than the existing estimators in terms of
scalability and flexibility. The output of MINE algorithm is not guaranteed to
be a consistent estimator. We propose a new estimator that instead of searching
among functions characterized by neural networks searches the functions in a
Reproducing Kernel Hilbert Space. We prove that the proposed estimator is
consistent. We carry out simulations and show that when the datasets are small
the proposed estimator is more reliable than the MINE estimator and when the
datasets are large the performance of the two methods are close.
