Fast gradient descent for drifting least squares regression, with
  application to bandits
  Online learning algorithms require to often recompute least squares
regression estimates of parameters. We study improving the computational
complexity of such algorithms by using stochastic gradient descent (SGD) type
schemes in place of classic regression solvers. We show that SGD schemes
efficiently track the true solutions of the regression problems, even in the
presence of a drift. This finding coupled with an $O(d)$ improvement in
complexity, where $d$ is the dimension of the data, make them attractive for
implementation in the big data settings. In the case when strong convexity in
the regression problem is guaranteed, we provide bounds on the error both in
expectation and high probability (the latter is often needed to provide
theoretical guarantees for higher level algorithms), despite the drifting least
squares solution. As an example of this case we prove that the regret
performance of an SGD version of the PEGE linear bandit algorithm
[Rusmevichientong and Tsitsiklis 2010] is worse that that of PEGE itself only
by a factor of $O(\log^4 n)$. When strong convexity of the regression problem
cannot be guaranteed, we investigate using an adaptive regularisation. We make
an empirical study of an adaptively regularised, SGD version of LinUCB [Li et
al. 2010] in a news article recommendation application, which uses the large
scale news recommendation dataset from Yahoo! front page. These experiments
show a large gain in computational complexity, with a consistently low tracking
error and click-through-rate (CTR) performance that is $75\%$ close.
