Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of
  Stochasticity
  Understanding the implicit bias of training algorithms is of crucial
importance in order to explain the success of overparametrised neural networks.
In this paper, we study the dynamics of stochastic gradient descent over
diagonal linear networks through its continuous time version, namely stochastic
gradient flow. We explicitly characterise the solution chosen by the stochastic
flow and prove that it always enjoys better generalisation properties than that
of gradient flow. Quite surprisingly, we show that the convergence speed of the
training loss controls the magnitude of the biasing effect: the slower the
convergence, the better the bias. To fully complete our analysis, we provide
convergence guarantees for the dynamics. We also give experimental results
which support our theoretical claims. Our findings highlight the fact that
structured noise can induce better generalisation and they help explain the
greater performances observed in practice of stochastic gradient descent over
gradient descent.
