CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
  Mutual information (MI) minimization has gained considerable interests in
various machine learning tasks. However, estimating and minimizing MI in
high-dimensional spaces remains a challenging problem, especially when only
samples, rather than distribution forms, are accessible. Previous works mainly
focus on MI lower bound approximation, which is not applicable to MI
minimization problems. In this paper, we propose a novel Contrastive Log-ratio
Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of
the properties of CLUB and its variational approximation. Based on this upper
bound, we introduce a MI minimization training scheme and further accelerate it
with a negative sampling strategy. Simulation studies on Gaussian distributions
show the reliable estimation ability of CLUB. Real-world MI minimization
experiments, including domain adaptation and information bottleneck,
demonstrate the effectiveness of the proposed method. The code is at
https://github.com/Linear95/CLUB.
