Particle Filtering Methods for Stochastic Optimization with Application
  to Large-Scale Empirical Risk Minimization
  This paper is concerned with sequential filtering based stochastic
optimization (FSO) approaches that leverage a probabilistic perspective to
implement the incremental proximity method (IPM). The present FSO methods are
derived based on the Kalman filter (KF) and the extended KF (EKF). In contrast
with typical methods such as stochastic gradient descent (SGD) and IPMs, they
do not need to pre-schedule the learning rate for convergence. Nevertheless,
they have limitations that inherit from the KF mechanism. As the particle
filtering (PF) method outperforms KF and its variants remarkably for nonlinear
non-Gaussian sequential filtering problems, it is natural to ask if FSO methods
can benefit from PF to get around of their limitations. We provide an
affirmative answer to this question by developing two PF based stochastic
optimizers (PFSOs). For performance evaluation, we apply them to address
nonlinear least-square fitting with simulated data, and empirical risk
minimization for binary classification of real data sets. Experimental results
demonstrate that PFSOs outperform remarkably a benchmark SGD algorithm, the
vanilla IPM, and KF-type FSO methods in terms of numerical stability,
convergence speed, and flexibility in handling diverse types of loss functions.
