Online Newton Step Algorithm with Estimated Gradient
  Online learning with limited information feedback (bandit) tries to solve the
problem where an online learner receives partial feedback information from the
environment in the course of learning. Under this setting, Flaxman et al.[8]
extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by
proposing the Online Gradient Descent with Expected Gradient (OGDEG) algorithm.
Specifically, it uses a simple trick to approximate the gradient of the loss
function $f_t$ by evaluating it at a single point and bounds the expected
regret as $\mathcal{O}(T^{5/6})$ [8], where the number of rounds is $T$.
Meanwhile, past research efforts have shown that compared with the first-order
algorithms, second-order online learning algorithms such as Online Newton Step
(ONS) [11] can significantly accelerate the convergence rate of traditional
online learning algorithms. Motivated by this, this paper aims to exploit the
second-order information to speed up the convergence of the OGDEG algorithm. In
particular, we extend the ONS algorithm with the trick of expected gradient and
develop a novel second-order online learning algorithm, i.e., Online Newton
Step with Expected Gradient (ONSEG). Theoretically, we show that the proposed
ONSEG algorithm significantly reduces the expected regret of OGDEG algorithm
from $\mathcal{O}(T^{5/6})$ to $\mathcal{O}(T^{2/3})$ in the bandit feedback
scenario. Empirically, we further demonstrate the advantages of the proposed
algorithm on multiple real-world datasets.
