Deforming the Loss Surface
  In deep learning, it is usually assumed that the shape of the loss surface is
fixed. Differently, a novel concept of deformation operator is first proposed
in this paper to deform the loss surface, thereby improving the optimization.
Deformation function, as a type of deformation operator, can improve the
generalization performance. Moreover, various deformation functions are
designed, and their contributions to the loss surface are further provided.
Then, the original stochastic gradient descent optimizer is theoretically
proved to be a flat minima filter that owns the talent to filter out the sharp
minima. Furthermore, the flatter minima could be obtained by exploiting the
proposed deformation functions, which is verified on CIFAR-100, with
visualizations of loss landscapes near the critical points obtained by both the
original optimizer and optimizer enhanced by deformation functions. The
experimental results show that deformation functions do find flatter regions.
Moreover, on ImageNet, CIFAR-10, and CIFAR-100, popular convolutional neural
networks enhanced by deformation functions are compared with the corresponding
original models, where significant improvements are observed on all of the
involved models equipped with deformation functions. For example, the top-1
test accuracy of ResNet-20 on CIFAR-100 increases by 1.46%, with insignificant
additional computational overhead.
