SPI-Optimizer: an integral-Separated PI Controller for Stochastic
  Optimization
  To overcome the oscillation problem in the classical momentum-based
optimizer, recent work associates it with the proportional-integral (PI)
controller, and artificially adds D term producing a PID controller. It
suppresses oscillation with the sacrifice of introducing extra hyper-parameter.
In this paper, we start by analyzing: why momentum-based method oscillates
about the optimal point? and answering that: the fluctuation problem relates to
the lag effect of integral (I) term. Inspired by the conditional integration
idea in classical control society, we propose SPI-Optimizer, an
integral-Separated PI controller based optimizer WITHOUT introducing extra
hyperparameter. It separates momentum term adaptively when the inconsistency of
current and historical gradient direction occurs. Extensive experiments
demonstrate that SPIOptimizer generalizes well on popular network architectures
to eliminate the oscillation, and owns competitive performance with faster
convergence speed (up to 40% epochs reduction ratio ) and more accurate
classification result on MNIST, CIFAR10, and CIFAR100 (up to 27.5% error
reduction ratio) than the state-of-the-art methods.
