On the Stability Properties and the Optimization Landscape of Training
  Problems with Squared Loss for Neural Networks and General Nonlinear Conic
  Approximation Schemes
  We study the optimization landscape and the stability properties of training
problems with squared loss for neural networks and general nonlinear conic
approximation schemes. It is demonstrated that, if a nonlinear conic
approximation scheme is considered that is (in an appropriately defined sense)
more expressive than a classical linear approximation approach and if there
exist unrealizable label vectors, then a training problem with squared loss is
necessarily unstable in the sense that its solution set depends discontinuously
on the label vector in the training data. We further prove that the same
effects that are responsible for these instability properties are also the
reason for the emergence of saddle points and spurious local minima, which may
be arbitrarily far away from global solutions, and that neither the instability
of the training problem nor the existence of spurious local minima can, in
general, be overcome by adding a regularization term to the objective function
that penalizes the size of the parameters in the approximation scheme. The
latter results are shown to be true regardless of whether the assumption of
realizability is satisfied or not. We demonstrate that our analysis in
particular applies to training problems for free-knot interpolation schemes and
deep and shallow neural networks with variable widths that involve an arbitrary
mixture of various activation functions (e.g., binary, sigmoid, tanh, arctan,
soft-sign, ISRU, soft-clip, SQNL, ReLU, leaky ReLU, soft-plus, bent identity,
SILU, ISRLU, and ELU). In summary, the findings of this paper illustrate that
the improved approximation properties of neural networks and general nonlinear
conic approximation instruments are linked in a direct and quantifiable way to
undesirable properties of the optimization problems that have to be solved in
order to train them.
