Dueling Bandits With Weak Regret
  We consider online content recommendation with implicit feedback through
pairwise comparisons, formalized as the so-called dueling bandit problem. We
study the dueling bandit problem in the Condorcet winner setting, and consider
two notions of regret: the more well-studied strong regret, which is 0 only
when both arms pulled are the Condorcet winner; and the less well-studied weak
regret, which is 0 if either arm pulled is the Condorcet winner. We propose a
new algorithm for this problem, Winner Stays (WS), with variations for each
kind of regret: WS for weak regret (WS-W) has expected cumulative weak regret
that is $O(N^2)$, and $O(N\log(N))$ if arms have a total order; WS for strong
regret (WS-S) has expected cumulative strong regret of $O(N^2 + N \log(T))$,
and $O(N\log(N)+N\log(T))$ if arms have a total order. WS-W is the first
dueling bandit algorithm with weak regret that is constant in time. WS is
simple to compute, even for problems with many arms, and we demonstrate through
numerical experiments on simulated and real data that WS has significantly
smaller regret than existing algorithms in both the weak- and strong-regret
settings.
