Finite-Time Analysis of Round-Robin Kullback-Leibler Upper Confidence
  Bounds for Optimal Adaptive Allocation with Multiple Plays and Markovian
  Rewards
  We study an extension of the classic stochastic multi-armed bandit problem
which involves multiple plays and Markovian rewards in the rested bandits
setting. In order to tackle this problem we consider an adaptive allocation
rule which at each stage combines the information from the sample means of all
the arms, with the Kullback-Leibler upper confidence bound of a single arm
which is selected in round-robin way. For rewards generated from a
one-parameter exponential family of Markov chains, we provide a finite-time
upper bound for the regret incurred from this adaptive allocation rule, which
reveals the logarithmic dependence of the regret on the time horizon, and which
is asymptotically optimal. For our analysis we devise several concentration
results for Markov chains, including a maximal inequality for Markov chains,
that may be of interest in their own right. As a byproduct of our analysis we
also establish asymptotically optimal, finite-time guarantees for the case of
multiple plays, and i.i.d. rewards drawn from a one-parameter exponential
family of probability densities. Additionally, we provide simulation results
that illustrate that calculating Kullback-Leibler upper confidence bounds in a
round-robin way, is significantly more efficient than calculating them for
every arm at each round, and that the expected regrets of those two approaches
behave similarly.
