Sharpness-Aware Minimization Improves Language Model Generalization
  The allure of superhuman-level capabilities has led to considerable interest
in language models like GPT-3 and T5, wherein the research has, by and large,
revolved around new model architectures, training tasks, and loss objectives,
along with substantial engineering efforts to scale up model capacity and
dataset size. Comparatively little work has been done to improve the
generalization of these models through better optimization. In this work, we
show that Sharpness-Aware Minimization (SAM), a recently proposed optimization
procedure that encourages convergence to flatter minima, can substantially
improve the generalization of language models without much computational
overhead. We show that SAM is able to boost performance on SuperGLUE, GLUE, Web
Questions, Natural Questions, Trivia QA, and TyDiQA, with particularly large
gains when training data for these tasks is limited.
