Regularized EM Algorithms: A Unified Framework and Statistical
  Guarantees
  Latent variable models are a fundamental modeling tool in machine learning
applications, but they present significant computational and analytical
challenges. The popular EM algorithm and its variants, is a much used
algorithmic tool; yet our rigorous understanding of its performance is highly
incomplete. Recently, work in Balakrishnan et al. (2014) has demonstrated that
for an important class of problems, EM exhibits linear local convergence. In
the high-dimensional setting, however, the M-step may not be well defined. We
address precisely this setting through a unified treatment using
regularization. While regularization for high-dimensional problems is by now
well understood, the iterative EM algorithm requires a careful balancing of
making progress towards the solution while identifying the right structure
(e.g., sparsity or low-rank). In particular, regularizing the M-step using the
state-of-the-art high-dimensional prescriptions (e.g., Wainwright (2014)) is
not guaranteed to provide this balance. Our algorithm and analysis are linked
in a way that reveals the balance between optimization and statistical errors.
We specialize our general framework to sparse gaussian mixture models,
high-dimensional mixed regression, and regression with missing variables,
obtaining statistical guarantees for each of these examples.
