A Caputo fractional derivative-based algorithm for optimization
  We propose a novel Caputo fractional derivative-based optimization algorithm.
Upon defining the Caputo fractional gradient with respect to the Cartesian
coordinate, we present a generic Caputo fractional gradient descent (CFGD)
method. We prove that the CFGD yields the steepest descent direction of a
locally smoothed objective function. The generic CFGD requires three parameters
to be specified, and a choice of the parameters yields a version of CFGD. We
propose three versions -- non-adaptive, adaptive terminal and adaptive order.
By focusing on quadratic objective functions, we provide a convergence
analysis. We prove that the non-adaptive CFGD converges to a Tikhonov
regularized solution. For the two adaptive versions, we derive error bounds,
which show convergence to integer-order stationary point under some conditions.
We derive an explicit formula of CFGD for quadratic functions. We
computationally found that the adaptive terminal (AT) CFGD mitigates the
dependence on the condition number in the rate of convergence and results in
significant acceleration over gradient descent (GD). For non-quadratic
functions, we develop an efficient implementation of CFGD using the
Gauss-Jacobi quadrature, whose computational cost is approximately proportional
to the number of the quadrature points and the cost of GD. Our numerical
examples show that AT-CFGD results in acceleration over GD, even when a small
number of the Gauss-Jacobi quadrature points (including a single point) is
used.
