AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with
  Swarm Intelligence
  This paper introduces AdaSwarm, a novel gradient-free optimizer which has
similar or even better performance than the Adam optimizer adopted in neural
networks. In order to support our proposed AdaSwarm, a novel Exponentially
weighted Momentum Particle Swarm Optimizer (EMPSO), is proposed. The ability of
AdaSwarm to tackle optimization problems is attributed to its capability to
perform good gradient approximations. We show that, the gradient of any
function, differentiable or not, can be approximated by using the parameters of
EMPSO. This is a novel technique to simulate GD which lies at the boundary
between numerical methods and swarm intelligence. Mathematical proofs of the
gradient approximation produced are also provided. AdaSwarm competes closely
with several state-of-the-art (SOTA) optimizers. We also show that AdaSwarm is
able to handle a variety of loss functions during backpropagation, including
the maximum absolute error (MAE).
