Convergence to Second-Order Stationarity for Non-negative Matrix
  Factorization: Provably and Concurrently
  Non-negative matrix factorization (NMF) is a fundamental non-convex
optimization problem with numerous applications in Machine Learning (music
analysis, document clustering, speech-source separation etc). Despite having
received extensive study, it is poorly understood whether or not there exist
natural algorithms that can provably converge to a local minimum. Part of the
reason is because the objective is heavily symmetric and its gradient is not
Lipschitz. In this paper we define a multiplicative weight update type dynamics
(modification of the seminal Lee-Seung algorithm) that runs concurrently and
provably avoids saddle points (first order stationary points that are not
second order). Our techniques combine tools from dynamical systems such as
stability and exploit the geometry of the NMF objective by reducing the
standard NMF formulation over the non-negative orthant to a new formulation
over (a scaled) simplex. An important advantage of our method is the use of
concurrent updates, which permits implementations in parallel computing
environments.
