Learning to Detect an Odd Restless Markov Arm with a Trembling Hand
  This paper studies the problem of finding an anomalous arm in a multi-armed
bandit when (a) each arm is a finite-state Markov process, and (b) the arms are
restless. Here, anomaly means that the transition probability matrix (TPM) of
one of the arms (the odd arm) is different from the common TPM of each of the
non-odd arms. The TPMs are unknown to a decision entity that wishes to find the
index of the odd arm as quickly as possible, subject to an upper bound on the
error probability. We derive a problem instance-specific asymptotic lower bound
on the expected time required to find the odd arm index, where the asymptotics
is as the error probability vanishes. Further, we devise a policy based on the
principle of certainty equivalence, and demonstrate that under a continuous
selection assumption and a certain regularity assumption on the TPMs, the
policy achieves the lower bound arbitrarily closely. Thus, while the lower
bound is shown for all problem instances, the upper bound is shown only for
those problem instances satisfying the continuous selection and the regularity
assumptions. Our achievability analysis is based on resolving the
identifiability problem in the context of a certain lifted countable-state
controlled Markov process.
