A Walk with SGD
  We present novel empirical observations regarding how stochastic gradient
descent (SGD) navigates the loss landscape of over-parametrized deep neural
networks (DNNs). These observations expose the qualitatively different roles of
learning rate and batch-size in DNN optimization and generalization.
Specifically we study the DNN loss surface along the trajectory of SGD by
interpolating the loss surface between parameters from consecutive
\textit{iterations} and tracking various metrics during training. We find that
the loss interpolation between parameters before and after each training
iteration's update is roughly convex with a minimum (\textit{valley floor}) in
between for most of the training. Based on this and other metrics, we deduce
that for most of the training update steps, SGD moves in valley like regions of
the loss surface by jumping from one valley wall to another at a height above
the valley floor. This 'bouncing between walls at a height' mechanism helps SGD
traverse larger distance for small batch sizes and large learning rates which
we find play qualitatively different roles in the dynamics. While a large
learning rate maintains a large height from the valley floor, a small batch
size injects noise facilitating exploration. We find this mechanism is crucial
for generalization because the valley floor has barriers and this exploration
above the valley floor allows SGD to quickly travel far away from the
initialization point (without being affected by barriers) and find flatter
regions, corresponding to better generalization.
