On the Loss Landscape of Adversarial Training: Identifying Challenges
  and How to Overcome Them
  We analyze the influence of adversarial training on the loss landscape of
machine learning models. To this end, we first provide analytical studies of
the properties of adversarial loss functions under different adversarial
budgets. We then demonstrate that the adversarial loss landscape is less
favorable to optimization, due to increased curvature and more scattered
gradients. Our conclusions are validated by numerical analyses, which show that
training under large adversarial budgets impede the escape from suboptimal
random initialization, cause non-vanishing gradients and make the model find
sharper minima. Based on these observations, we show that a periodic
adversarial scheduling (PAS) strategy can effectively overcome these
challenges, yielding better results than vanilla adversarial training while
being much less sensitive to the choice of learning rate.
