A Closer Look at Deep Learning Heuristics: Learning rate restarts,
  Warmup and Distillation
  The convergence rate and final performance of common deep learning models
have significantly benefited from heuristics such as learning rate schedules,
knowledge distillation, skip connections, and normalization layers. In the
absence of theoretical underpinnings, controlled experiments aimed at
explaining these strategies can aid our understanding of deep learning
landscapes and the training dynamics. Existing approaches for empirical
analysis rely on tools of linear interpolation and visualizations with
dimensionality reduction, each with their limitations. Instead, we revisit such
analysis of heuristics through the lens of recently proposed methods for loss
surface and representation analysis, viz., mode connectivity and canonical
correlation analysis (CCA), and hypothesize reasons for the success of the
heuristics. In particular, we explore knowledge distillation and learning rate
heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our
empirical analysis suggests that: (a) the reasons often quoted for the success
of cosine annealing are not evidenced in practice; (b) that the effect of
learning rate warmup is to prevent the deeper layers from creating training
instability; and (c) that the latent knowledge shared by the teacher is
primarily disbursed to the deeper layers.
