diffGrad: An Optimization Method for Convolutional Neural Networks
  Stochastic Gradient Decent (SGD) is one of the core techniques behind the
success of deep neural networks. The gradient provides information on the
direction in which a function has the steepest rate of change. The main problem
with basic SGD is to change by equal sized steps for all parameters,
irrespective of gradient behavior. Hence, an efficient way of deep network
optimization is to make adaptive step sizes for each parameter. Recently,
several attempts have been made to improve gradient descent methods such as
AdaGrad, AdaDelta, RMSProp and Adam. These methods rely on the square roots of
exponential moving averages of squared past gradients. Thus, these methods do
not take advantage of local change in gradients. In this paper, a novel
optimizer is proposed based on the difference between the present and the
immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization
technique, the step size is adjusted for each parameter in such a way that it
should have a larger step size for faster gradient changing parameters and a
lower step size for lower gradient changing parameters. The convergence
analysis is done using the regret bound approach of online learning framework.
Rigorous analysis is made in this paper over three synthetic complex non-convex
functions. The image categorization experiments are also conducted over the
CIFAR10 and CIFAR100 datasets to observe the performance of diffGrad with
respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta,
RMSProp, AMSGrad, and Adam. The residual unit (ResNet) based Convolutional
Neural Networks (CNN) architecture is used in the experiments. The experiments
show that diffGrad outperforms other optimizers. Also, we show that diffGrad
performs uniformly well for training CNN using different activation functions.
The source code is made publicly available at
https://github.com/shivram1987/diffGrad.
