A Fair Comparison of Two Popular Flat Minima Optimizers: Stochastic
  Weight Averaging vs. Sharpness-Aware Minimization
  Recently, flat-minima optimizers, which seek to find parameters in low loss
neighborhoods, have been shown to improve upon stochastic and adaptive
gradient-based optimizers for training neural networks. Two methods have
received significant attention due to their impressive generalization
performance and scalability: 1. Stochastic Weight Averaging (SWA), and 2.
Sharpness Aware Minimization (SAM). However, there has been limited
investigation into their properties and no systematic benchmarking of them.
Previous work mainly evaluated SWA and SAM on different architectures and
datasets. We fill this gap here by comparing the loss surfaces of the models
trained with each method and through a broad benchmarking across computer
vision, natural language processing, and graph representation learning tasks.
We discover a number of surprising findings from these results, which we hope
will help researchers further improve deep learning optimizers, and
practitioners identify the right optimizer for their problem.
