AngularGrad: A New Optimization Technique for Angular Convergence of
  Convolutional Neural Networks
  Convolutional neural networks (CNNs) are trained using stochastic gradient
descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam)
optimizer has become very popular due to its adaptive momentum, which tackles
the dying gradient problem of SGD. Nevertheless, existing optimizers are still
unable to exploit the optimization curvature information efficiently. This
paper proposes a new AngularGrad optimizer that considers the behavior of the
direction/angle of consecutive gradients. This is the first attempt in the
literature to exploit the gradient angular information apart from its
magnitude. The proposed AngularGrad generates a score to control the step size
based on the gradient angular information of previous iterations. Thus, the
optimization steps become smoother as a more accurate step size of immediate
past gradients is captured through the angular information. Two variants of
AngularGrad are developed based on the use of Tangent or Cosine functions for
computing the gradient angular information. Theoretically, AngularGrad exhibits
the same regret bound as Adam for convergence purposes. Nevertheless, extensive
experiments conducted on benchmark data sets against state-of-the-art methods
reveal a superior performance of AngularGrad. The source code will be made
publicly available at: https://github.com/mhaut/AngularGrad.
