On the Behavior of the Expectation-Maximization Algorithm for Mixture
  Models
  Finite mixture models are among the most popular statistical models used in
different data science disciplines. Despite their broad applicability,
inference under these models typically leads to computationally challenging
non-convex problems. While the Expectation-Maximization (EM) algorithm is the
most popular approach for solving these non-convex problems, the behavior of
this algorithm is not well understood. In this work, we focus on the case of
mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple
equally weighted mixture of two single dimensional Laplacian distributions and
show that every local optimum of the population maximum likelihood estimation
problem is globally optimal. Then, we prove that the EM algorithm converges to
the ground truth parameters almost surely with random initialization. Our
result extends the existing results for Gaussian distribution to Laplacian
distribution. Then we numerically study the behavior of mixture models with
more than two components. Motivated by our extensive numerical experiments, we
propose a novel stochastic method for estimating the mean of components of a
mixture model. Our numerical experiments show that our algorithm outperforms
the Naive EM algorithm in almost all scenarios.
