Penalizing Gradient Norm for Efficiently Improving Generalization in
  Deep Learning
  How to train deep neural networks (DNNs) to generalize well is a central
concern in deep learning, especially for severely overparameterized networks
nowadays. In this paper, we propose an effective method to improve the model
generalization by additionally penalizing the gradient norm of loss function
during optimization. We demonstrate that confining the gradient norm of loss
function could help lead the optimizers towards finding flat minima. We
leverage the first-order approximation to efficiently implement the
corresponding gradient to fit well in the gradient descent framework. In our
experiments, we confirm that when using our methods, generalization performance
of various models could be improved on different datasets. Also, we show that
the recent sharpness-aware minimization method \cite{DBLP:conf/iclr/ForetKMN21}
is a special, but not the best, case of our method, where the best case of our
method could give new state-of-art performance on these tasks.
