Stochastic Gradient Variance Reduction by Solving a Filtering Problem
  Deep neural networks (DNN) are typically optimized using stochastic gradient
descent (SGD). However, the estimation of the gradient using stochastic samples
tends to be noisy and unreliable, resulting in large gradient variance and bad
convergence. In this paper, we propose \textbf{Filter Gradient Decent}~(FGD),
an efficient stochastic optimization algorithm that makes the consistent
estimation of the local gradient by solving an adaptive filtering problem with
different design of filters. Our method reduces variance in stochastic gradient
descent by incorporating the historical states to enhance the current
estimation. It is able to correct noisy gradient direction as well as to
accelerate the convergence of learning. We demonstrate the effectiveness of the
proposed Filter Gradient Descent on numerical optimization and training neural
networks, where it achieves superior and robust performance compared with
traditional momentum-based methods. To the best of our knowledge, we are the
first to provide a practical solution that integrates filtering into gradient
estimation by making the analogy between gradient estimation and filtering
problems in signal processing. (The code is provided in
https://github.com/Adamdad/Filter-Gradient-Decent)
