A Loss Curvature Perspective on Training Instability in Deep Learning
  In this work, we study the evolution of the loss Hessian across many
classification tasks in order to understand the effect the curvature of the
loss has on the training dynamics. Whereas prior work has focused on how
different learning rates affect the loss Hessian observed during training, we
also analyze the effects of model initialization, architectural choices, and
common training heuristics such as gradient clipping and learning rate warmup.
Our results demonstrate that successful model and hyperparameter choices allow
the early optimization trajectory to either avoid -- or navigate out of --
regions of high curvature and into flatter regions that tolerate a higher
learning rate. Our results suggest a unifying perspective on how disparate
mitigation strategies for training instability ultimately address the same
underlying failure mode of neural network optimization, namely poor
conditioning. Inspired by the conditioning perspective, we show that learning
rate warmup can improve training stability just as much as batch normalization,
layer normalization, MetaInit, GradInit, and Fixup initialization.
