Multi-kernel Passive Stochastic Gradient Algorithms and Transfer
  Learning
  This paper develops a novel passive stochastic gradient algorithm. In passive
stochastic approximation, the stochastic gradient algorithm does not have
control over the location where noisy gradients of the cost function are
evaluated. Classical passive stochastic gradient algorithms use a kernel that
approximates a Dirac delta to weigh the gradients based on how far they are
evaluated from the desired point. In this paper we construct a multi-kernel
passive stochastic gradient algorithm. The algorithm performs substantially
better in high dimensional problems and incorporates variance reduction. We
analyze the weak convergence of the multi-kernel algorithm and its rate of
convergence. In numerical examples, we study the multi-kernel version of the
passive least mean squares (LMS) algorithm for transfer learning to compare the
performance with the classical passive version.
