Improving Model Training by Periodic Sampling over Weight Distributions
  In this paper, we explore techniques centered around periodic sampling of
model weights that provide convergence improvements on gradient update methods
(vanilla \acs{SGD}, Momentum, Adam) for a variety of vision problems
(classification, detection, segmentation). Importantly, our algorithms provide
better, faster and more robust convergence and training performance with only a
slight increase in computation time. Our techniques are independent of the
neural network model, gradient optimization methods or existing optimal
training policies and converge in a less volatile fashion with performance
improvements that are approximately monotonic. We conduct a variety of
experiments to quantify these improvements and identify scenarios where these
techniques could be more useful.
