Theoretical properties of the global optimizer of two layer neural
  network
  In this paper, we study the problem of optimizing a two-layer artificial
neural network that best fits a training dataset. We look at this problem in
the setting where the number of parameters is greater than the number of
sampled points. We show that for a wide class of differentiable activation
functions (this class involves "almost" all functions which are not piecewise
linear), we have that first-order optimal solutions satisfy global optimality
provided the hidden layer is non-singular. Our results are easily extended to
hidden layers given by a flat matrix from that of a square matrix. Results are
applicable even if network has more than one hidden layer provided all hidden
layers satisfy non-singularity, all activations are from the given "good" class
of differentiable functions and optimization is only with respect to the last
hidden layer. We also study the smoothness properties of the objective function
and show that it is actually Lipschitz smooth, i.e., its gradients do not
change sharply. We use smoothness properties to guarantee asymptotic
convergence of O(1/number of iterations) to a first-order optimal solution. We
also show that our algorithm will maintain non-singularity of hidden layer for
any finite number of iterations.
