Regret Bounds for Restless Markov Bandits
  We consider the restless Markov bandit problem, in which the state of each
arm evolves according to a Markov process independently of the learner's
actions. We suggest an algorithm that after $T$ steps achieves
$\tilde{O}(\sqrt{T})$ regret with respect to the best policy that knows the
distributions of all arms. No assumptions on the Markov chains are made except
that they are irreducible. In addition, we show that index-based policies are
necessarily suboptimal for the considered problem.
