The Landscape of Multi-Layer Linear Neural Network From the Perspective
  of Algebraic Geometry
  The clear understanding of the non-convex landscape of neural network is a
complex incomplete problem. This paper studies the landscape of linear
(residual) network, the simplified version of the nonlinear network. By
treating the gradient equations as polynomial equations, we use algebraic
geometry tools to solve it over the complex number field, the attained solution
can be decomposed into different irreducible complex geometry objects. Then
three hypotheses are proposed, involving how to calculate the loss on each
irreducible geometry object, the losses of critical points have a certain range
and the relationship between the dimension of each irreducible geometry object
and strict saddle condition. Finally, numerical algebraic geometry is applied
to verify the rationality of these three hypotheses which further clarify the
landscape of linear network and the role of residual connection.
