Generalized Majorization-Minimization
  Non-convex optimization is ubiquitous in machine learning.
Majorization-Minimization (MM) is a powerful iterative procedure for optimizing
non-convex functions that works by optimizing a sequence of bounds on the
function. In MM, the bound at each iteration is required to \emph{touch} the
objective function at the optimizer of the previous bound. We show that this
touching constraint is unnecessary and overly restrictive. We generalize MM by
relaxing this constraint, and propose a new optimization framework, named
Generalized Majorization-Minimization (G-MM), that is more flexible. For
instance, G-MM can incorporate application-specific biases into the
optimization procedure without changing the objective function. We derive G-MM
algorithms for several latent variable models and show empirically that they
consistently outperform their MM counterparts in optimizing non-convex
objectives. In particular, G-MM algorithms appear to be less sensitive to
initialization.
