The Transitive Information Theory and its Application to Deep Generative
  Models
  Paradoxically, a Variational Autoencoder (VAE) could be pushed in two
opposite directions, utilizing powerful decoder model for generating realistic
images but collapsing the learned representation, or increasing regularization
coefficient for disentangling representation but ultimately generating blurry
examples. Existing methods narrow the issues to the rate-distortion trade-off
between compression and reconstruction. We argue that a good reconstruction
model does learn high capacity latents that encode more details, however, its
use is hindered by two major issues: the prior is random noise which is
completely detached from the posterior and allow no controllability in the
generation; mean-field variational inference doesn't enforce hierarchy
structure which makes the task of recombining those units into plausible novel
output infeasible. As a result, we develop a system that learns a hierarchy of
disentangled representation together with a mechanism for recombining the
learned representation for generalization. This is achieved by introducing a
minimal amount of inductive bias to learn controllable prior for the VAE. The
idea is supported by here developed transitive information theory, that is, the
mutual information between two target variables could alternately be maximized
through the mutual information to the third variable, thus bypassing the
rate-distortion bottleneck in VAE design. In particular, we show that our
model, named SemafoVAE (inspired by the similar concept in computer science),
could generate high-quality examples in a controllable manner, perform smooth
traversals of the disentangled factors and intervention at a different level of
representation hierarchy.
