Benefits of over-parameterization with EM
  Expectation Maximization (EM) is among the most popular algorithms for
maximum likelihood estimation, but it is generally only guaranteed to find its
stationary points of the log-likelihood objective. The goal of this article is
to present theoretical and empirical evidence that over-parameterization can
help EM avoid spurious local optima in the log-likelihood. We consider the
problem of estimating the mean vectors of a Gaussian mixture model in a
scenario where the mixing weights are known. Our study shows that the global
behavior of EM, when one uses an over-parameterized model in which the mixing
weights are treated as unknown, is better than that when one uses the (correct)
model with the mixing weights fixed to the known values. For symmetric
Gaussians mixtures with two components, we prove that introducing the
(statistically redundant) weight parameters enables EM to find the global
maximizer of the log-likelihood starting from almost any initial mean
parameters, whereas EM without this over-parameterization may very often fail.
For other Gaussian mixtures, we provide empirical evidence that shows similar
behavior. Our results corroborate the value of over-parameterization in solving
non-convex optimization problems, previously observed in other domains.
