Understanding the Generalization of Adam in Learning Neural Networks
  with Proper Regularization
  Adaptive gradient methods such as Adam have gained increasing popularity in
deep learning optimization. However, it has been observed that compared with
(stochastic) gradient descent, Adam can converge to a different solution with a
significantly worse test error in many deep learning applications such as image
classification, even with a fine-tuned regularization. In this paper, we
provide a theoretical explanation for this phenomenon: we show that in the
nonconvex setting of learning over-parameterized two-layer convolutional neural
networks starting from the same random initialization, for a class of data
distributions (inspired from image data), Adam and gradient descent (GD) can
converge to different global solutions of the training objective with provably
different generalization errors, even with weight decay regularization. In
contrast, we show that if the training objective is convex, and the weight
decay regularization is employed, any optimization algorithms including Adam
and GD will converge to the same solution if the training is successful. This
suggests that the inferior generalization performance of Adam is fundamentally
tied to the nonconvex landscape of deep learning optimization.
