Over-Parametrized Matrix Factorization in the Presence of Spurious
  Stationary Points
  Motivated by the emerging role of interpolating machines in signal processing
and machine learning, this work considers the computational aspects of
over-parametrized matrix factorization. In this context, the optimization
landscape may contain spurious stationary points (SSPs), which are proved to be
full-rank matrices. The presence of these SSPs means that it is impossible to
hope for any global guarantees in over-parametrized matrix factorization. For
example, when initialized at an SSP, the gradient flow will be trapped there
forever. Nevertheless, despite these SSPs, we establish in this work that the
gradient flow of the corresponding merit function converges to a global
minimizer, provided that its initialization is rank-deficient and sufficiently
close to the feasible set of the optimization problem. We numerically observe
that a heuristic discretization of the proposed gradient flow, inspired by
primal-dual algorithms, is successful when initialized randomly. Our result is
in sharp contrast with the local refinement methods which require an
initialization close to the optimal set of the optimization problem. More
specifically, we successfully avoid the traps set by the SSPs because the
gradient flow remains rank-deficient at all times, and not because there are no
SSPs nearby. The latter is the case for the local refinement methods. Moreover,
the widely-used restricted isometry property plays no role in our main result.
