Convergence and Implicit Bias of Gradient Flow on Overparametrized
  Linear Networks
  Neural networks trained via gradient descent with random initialization and
without any regularization enjoy good generalization performance in practice
despite being highly overparametrized. A promising direction to explain this
phenomenon is to study how initialization and overparametrization affect
convergence and implicit bias of training algorithms. In this paper, we present
a novel analysis of single-hidden-layer linear networks trained under gradient
flow, which connects initialization, optimization, and overparametrization.
Firstly, we show that the squared loss converges exponentially to its optimum
at a rate that depends on the level of imbalance and the margin of the
initialization. Secondly, we show that proper initialization constrains the
dynamics of the network parameters to lie within an invariant set. In turn,
minimizing the loss over this set leads to the min-norm solution. Finally, we
show that large hidden layer width, together with (properly scaled) random
initialization, ensures proximity to such an invariant set during training,
allowing us to derive a novel non-asymptotic upper-bound on the distance
between the trained network and the min-norm solution.
