Path Regularization: A Convexity and Sparsity Inducing Regularization
  for Parallel ReLU Networks
  Understanding the fundamental principles behind the success of deep neural
networks is one of the most important open questions in the current literature.
To this end, we study the training problem of deep neural networks and
introduce an analytic approach to unveil hidden convexity in the optimization
landscape. We consider a deep parallel ReLU network architecture, which also
includes standard deep networks and ResNets as its special cases. We then show
that pathwise regularized training problems can be represented as an exact
convex optimization problem. We further prove that the equivalent convex
problem is regularized via a group sparsity inducing norm. Thus, a path
regularized parallel ReLU network can be viewed as a parsimonious convex model
in high dimensions. More importantly, we show that the computational complexity
required to globally optimize the equivalent convex problem is fully
polynomial-time in feature dimension and number of samples. Therefore, we prove
polynomial-time trainability of path regularized ReLU networks with global
optimality guarantees. We also provide several numerical experiments
corroborating our theory.
