Deforming the Loss Surface to Affect the Behaviour of the Optimizer
  In deep learning, it is usually assumed that the optimization process is
conducted on a shape-fixed loss surface. Differently, we first propose a novel
concept of deformation mapping in this paper to affect the behaviour of the
optimizer. Vertical deformation mapping (VDM), as a type of deformation
mapping, can make the optimizer enter a flat region, which often implies better
generalization performance. Moreover, we design various VDMs, and further
provide their contributions to the loss surface. After defining the local M
region, theoretical analyses show that deforming the loss surface can enhance
the gradient descent optimizer's ability to filter out sharp minima. With
visualizations of loss landscapes, we evaluate the flatnesses of minima
obtained by both the original optimizer and optimizers enhanced by VDMs on
CIFAR-100. The experimental results show that VDMs do find flatter regions.
Moreover, we compare popular convolutional neural networks enhanced by VDMs
with the corresponding original ones on ImageNet, CIFAR-10, and CIFAR-100. The
results are surprising: there are significant improvements on all of the
involved models equipped with VDMs. For example, the top-1 test accuracy of
ResNet-20 on CIFAR-100 increases by 1.46%, with insignificant additional
computational overhead.
