Thompson Sampling in Non-Episodic Restless Bandits
  Restless bandit problems assume time-varying reward distributions of the
arms, which adds flexibility to the model but makes the analysis more
challenging. We study learning algorithms over the unknown reward distributions
and prove a sub-linear, $O(\sqrt{T}\log T)$, regret bound for a variant of
Thompson sampling. Our analysis applies in the infinite time horizon setting,
resolving the open question raised by Jung and Tewari (2019) whose analysis is
limited to the episodic case. We adopt their policy mapping framework, which
allows our algorithm to be efficient and simultaneously keeps the regret
meaningful. Our algorithm adapts the TSDE algorithm of Ouyang et al. (2017) in
a non-trivial manner to account for the special structure of restless bandits.
We test our algorithm on a simulated dynamic channel access problem with
several policy mappings, and the empirical regrets agree with the theoretical
bound regardless of the choice of the policy mapping.
