The Break-Even Point on Optimization Trajectories of Deep Neural
  Networks
  The early phase of training of deep neural networks is critical for their
final performance. In this work, we study how the hyperparameters of stochastic
gradient descent (SGD) used in the early phase of training affect the rest of
the optimization trajectory. We argue for the existence of the "break-even"
point on this trajectory, beyond which the curvature of the loss surface and
noise in the gradient are implicitly regularized by SGD. In particular, we
demonstrate on multiple classification tasks that using a large learning rate
in the initial phase of training reduces the variance of the gradient, and
improves the conditioning of the covariance of gradients. These effects are
beneficial from the optimization perspective and become visible after the
break-even point. Complementing prior work, we also show that using a low
learning rate results in bad conditioning of the loss surface even for a neural
network with batch normalization layers. In short, our work shows that key
properties of the loss surface are strongly influenced by SGD in the early
phase of training. We argue that studying the impact of the identified effects
on generalization is a promising future direction.
