Best Arm Identification in Restless Markov Multi-Armed Bandits
  We study the problem of identifying the best arm in a multi-armed bandit
environment when each arm is a time-homogeneous and ergodic discrete-time
Markov process on a common, finite state space. The state evolution on each arm
is governed by the arm's transition probability matrix (TPM). A decision entity
that knows the set of arm TPMs but not the exact mapping of the TPMs to the
arms, wishes to find the index of the best arm as quickly as possible, subject
to an upper bound on the error probability. The decision entity selects one arm
at a time sequentially, and all the unselected arms continue to undergo state
evolution ({\em restless} arms). For this problem, we derive the first-known
problem instance-dependent asymptotic lower bound on the growth rate of the
expected time required to find the index of the best arm, where the asymptotics
is as the error probability vanishes. Further, we propose a sequential policy
that, for an input parameter $R$, forcibly selects an arm that has not been
selected for $R$ consecutive time instants. We show that this policy achieves
an upper bound that depends on $R$ and is monotonically non-increasing as
$R\to\infty$. The question of whether, in general, the limiting value of the
upper bound as $R\to\infty$ matches with the lower bound, remains open. We
identify a special case in which the upper and the lower bounds match. Prior
works on best arm identification have dealt with (a) independent and
identically distributed observations from the arms, and (b) rested Markov arms,
whereas our work deals with the more difficult setting of restless Markov arms.
