High Mutual Information in Representation Learning with Symmetric
  Variational Inference
  We introduce the Mutual Information Machine (MIM), a novel formulation of
representation learning, using a joint distribution over the observations and
latent state in an encoder/decoder framework. Our key principles are symmetry
and mutual information, where symmetry encourages the encoder and decoder to
learn different factorizations of the same underlying distribution, and mutual
information, to encourage the learning of useful representations for downstream
tasks. Our starting point is the symmetric Jensen-Shannon divergence between
the encoding and decoding joint distributions, plus a mutual information
encouraging regularizer. We show that this can be bounded by a tractable cross
entropy loss function between the true model and a parameterized approximation,
and relate this to the maximum likelihood framework. We also relate MIM to
variational autoencoders (VAEs) and demonstrate that MIM is capable of learning
symmetric factorizations, with high mutual information that avoids posterior
collapse.
