Applications of Common Entropy for Causal Inference
  We study the problem of discovering the simplest latent variable that can
make two observed discrete variables conditionally independent. The minimum
entropy required for such a latent is known as common entropy in information
theory. We extend this notion to Renyi common entropy by minimizing the Renyi
entropy of the latent variable. To efficiently compute common entropy, we
propose an iterative algorithm that can be used to discover the trade-off
between the entropy of the latent variable and the conditional mutual
information of the observed variables. We show two applications of common
entropy in causal inference: First, under the assumption that there are no
low-entropy mediators, it can be used to distinguish causation from spurious
correlation among almost all joint distributions on simple causal graphs with
two observed variables. Second, common entropy can be used to improve
constraint-based methods such as PC or FCI algorithms in the small-sample
regime, where these methods are known to struggle. We propose a modification to
these constraint-based methods to assess if a separating set found by these
algorithms is valid using common entropy. We finally evaluate our algorithms on
synthetic and real data to establish their performance.
