Be Greedy in Multi-Armed Bandits
  The Greedy algorithm is the simplest heuristic in sequential decision problem
that carelessly takes the locally optimal choice at each round, disregarding
any advantages of exploring and/or information gathering. Theoretically, it is
known to sometimes have poor performances, for instance even a linear regret
(with respect to the time horizon) in the standard multi-armed bandit problem.
On the other hand, this heuristic performs reasonably well in practice and it
even has sublinear, and even near-optimal, regret bounds in some very specific
linear contextual and Bayesian bandit models. We build on a recent line of work
and investigate bandit settings where the number of arms is relatively large
and where simple greedy algorithms enjoy highly competitive performance, both
in theory and in practice. We first provide a generic worst-case bound on the
regret of the Greedy algorithm. When combined with some arms subsampling, we
prove that it verifies near-optimal worst-case regret bounds in continuous,
infinite and many-armed bandit problems. Moreover, for shorter time spans, the
theoretical relative suboptimality of Greedy is even reduced. As a consequence,
we subversively claim that for many interesting problems and associated
horizons, the best compromise between theoretical guarantees, practical
performances and computational burden is definitely to follow the greedy
heuristic. We support our claim by many numerical experiments that show
significant improvements compared to the state-of-the-art, even for moderately
long time horizon.
