Comparing EM with GD in Mixture Models of Two Components
  The expectation-maximization (EM) algorithm has been widely used in
minimizing the negative log likelihood (also known as cross entropy) of mixture
models. However, little is understood about the goodness of the fixed points it
converges to. In this paper, we study the regions where one component is
missing in two-component mixture models, which we call one-cluster regions. We
analyze the propensity of such regions to trap EM and gradient descent (GD) for
mixtures of two Gaussians and mixtures of two Bernoullis. In the case of
Gaussian mixtures, EM escapes one-cluster regions exponentially fast, while GD
escapes them linearly fast. In the case of mixtures of Bernoullis, we find that
there exist one-cluster regions that are stable for GD and therefore trap GD,
but those regions are unstable for EM, allowing EM to escape. Those regions are
local minima that appear universally in experiments and can be arbitrarily bad.
This work implies that EM is less likely than GD to converge to certain bad
local optima in mixture models.
