Online Model Selection: a Rested Bandit Formulation
  Motivated by a natural problem in online model selection with bandit
information, we introduce and analyze a best arm identification problem in the
rested bandit setting, wherein arm expected losses decrease with the number of
times the arm has been played. The shape of the expected loss functions is
similar across arms, and is assumed to be available up to unknown parameters
that have to be learned on the fly. We define a novel notion of regret for this
problem, where we compare to the policy that always plays the arm having the
smallest expected loss at the end of the game. We analyze an arm elimination
algorithm whose regret vanishes as the time horizon increases. The actual rate
of convergence depends in a detailed way on the postulated functional form of
the expected losses. Unlike known model selection efforts in the recent bandit
literature, our algorithm exploits the specific structure of the problem to
learn the unknown parameters of the expected loss function so as to identify
the best arm as quickly as possible. We complement our analysis with a lower
bound, indicating strengths and limitations of the proposed solution.
