How regularization affects the critical points in linear networks
  This paper is concerned with the problem of representing and learning a
linear transformation using a linear neural network. In recent years, there has
been a growing interest in the study of such networks in part due to the
successes of deep learning. The main question of this body of research and also
of this paper pertains to the existence and optimality properties of the
critical points of the mean-squared loss function. The primary concern here is
the robustness of the critical points with regularization of the loss function.
An optimal control model is introduced for this purpose and a learning
algorithm (regularized form of backprop) derived for the same using the
Hamilton's formulation of optimal control. The formulation is used to provide a
complete characterization of the critical points in terms of the solutions of a
nonlinear matrix-valued equation, referred to as the characteristic equation.
Analytical and numerical tools from bifurcation theory are used to compute the
critical points via the solutions of the characteristic equation. The main
conclusion is that the critical point diagram can be fundamentally different
even with arbitrary small amounts of regularization.
