Exploiting Chain Rule and Bayes' Theorem to Compare Probability
  Distributions
  To measure the difference between two probability distributions, referred to
as the source and target, respectively, we exploit both the chain rule and
Bayes' theorem to construct conditional transport (CT), which is constituted by
both a forward component and a backward one. The forward CT is the expected
cost of moving a source data point to a target one, with their joint
distribution defined by the product of the source probability density function
(PDF) and a source-dependent conditional distribution, which is related to the
target PDF via Bayes' theorem. The backward CT is defined by reversing the
direction. The CT cost can be approximated by replacing the source and target
PDFs with their discrete empirical distributions supported on mini-batches,
making it amenable to implicit distributions and stochastic gradient
descent-based optimization. When applied to train a generative model, CT is
shown to strike a good balance between mode-covering and mode-seeking behaviors
and strongly resist mode collapse. On a wide variety of benchmark datasets for
generative modeling, substituting the default statistical distance of an
existing generative adversarial network with CT is shown to consistently
improve the performance. PyTorch code is provided.
