Global Optimality in Tensor Factorization, Deep Learning, and Beyond
  Techniques involving factorization are found in a wide range of applications
and have enjoyed significant empirical success in many fields. However, common
to a vast majority of these problems is the significant disadvantage that the
associated optimization problems are typically non-convex due to a multilinear
form or other convexity destroying transformation. Here we build on ideas from
convex relaxations of matrix factorizations and present a very general
framework which allows for the analysis of a wide range of non-convex
factorization problems - including matrix factorization, tensor factorization,
and deep neural network training formulations. We derive sufficient conditions
to guarantee that a local minimum of the non-convex optimization problem is a
global minimum and show that if the size of the factorized variables is large
enough then from any initialization it is possible to find a global minimizer
using a purely local descent algorithm. Our framework also provides a partial
theoretical justification for the increasingly common use of Rectified Linear
Units (ReLUs) in deep neural networks and offers guidance on deep network
architectures and regularization strategies to facilitate efficient
optimization.
