Expected Information Maximization: Using the I-Projection for Mixture
  Density Estimation
  Modelling highly multi-modal data is a challenging problem in machine
learning. Most algorithms are based on maximizing the likelihood, which
corresponds to the M(oment)-projection of the data distribution to the model
distribution. The M-projection forces the model to average over modes it cannot
represent. In contrast, the I(information)-projection ignores such modes in the
data and concentrates on the modes the model can represent. Such behavior is
appealing whenever we deal with highly multi-modal data where modelling single
modes correctly is more important than covering all the modes. Despite this
advantage, the I-projection is rarely used in practice due to the lack of
algorithms that can efficiently optimize it based on data. In this work, we
present a new algorithm called Expected Information Maximization (EIM) for
computing the I-projection solely based on samples for general latent variable
models, where we focus on Gaussian mixtures models and Gaussian mixtures of
experts. Our approach applies a variational upper bound to the I-projection
objective which decomposes the original objective into single objectives for
each mixture component as well as for the coefficients, allowing an efficient
optimization. Similar to GANs, our approach employs discriminators but uses a
more stable optimization procedure, using a tight upper bound. We show that our
algorithm is much more effective in computing the I-projection than recent GAN
approaches and we illustrate the effectiveness of our approach for modelling
multi-modal behavior on two pedestrian and traffic prediction datasets.
