Understanding the unstable convergence of gradient descent
  Most existing analyses of (stochastic) gradient descent rely on the condition
that for $L$-smooth costs, the step size is less than $2/L$. However, many
works have observed that in machine learning applications step sizes often do
not fulfill this condition, yet (stochastic) gradient descent still converges,
albeit in an unstable manner. We investigate this unstable convergence
phenomenon from first principles, and discuss key causes behind it. We also
identify its main characteristics, and how they interrelate based on both
theory and experiments, offering a principled view toward understanding the
phenomenon.
