Global Convergence Analysis of Deep Linear Networks with A One-neuron
  Layer
  In this paper, we follow Eftekhari's work to give a non-local convergence
analysis of deep linear networks. Specifically, we consider optimizing deep
linear networks which have a layer with one neuron under quadratic loss. We
describe the convergent point of trajectories with arbitrary starting point
under gradient flow, including the paths which converge to one of the saddle
points or the original point. We also show specific convergence rates of
trajectories that converge to the global minimizer by stages. To achieve these
results, this paper mainly extends the machinery in Eftekhari's work to
provably identify the rank-stable set and the global minimizer convergent set.
We also give specific examples to show the necessity of our definitions.
Crucially, as far as we know, our results appear to be the first to give a
non-local global analysis of linear neural networks from arbitrary initialized
points, rather than the lazy training regime which has dominated the literature
of neural networks, and restricted benign initialization in Eftekhari's work.
We also note that extending our results to general linear networks without one
hidden neuron assumption remains a challenging open problem.
