Regularizing Neural Networks via Adversarial Model Perturbation
  Effective regularization techniques are highly desired in deep learning for
alleviating overfitting and improving generalization. This work proposes a new
regularization scheme, based on the understanding that the flat local minima of
the empirical risk cause the model to generalize better. This scheme is
referred to as adversarial model perturbation (AMP), where instead of directly
minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD.
Specifically, the AMP loss is obtained from the empirical risk by applying the
"worst" norm-bounded perturbation on each point in the parameter space.
Comparing with most existing regularization schemes, AMP has strong theoretical
justifications, in that minimizing the AMP loss can be shown theoretically to
favour flat local minima of the empirical risk. Extensive experiments on
various modern deep architectures establish AMP as a new state of the art among
regularization schemes. Our code is available at
https://github.com/hiyouga/AMP-Regularizer.
