Nonstationary Stochastic Multiarmed Bandits: UCB Policies and Minimax
  Regret
  We study the nonstationary stochastic Multi-Armed Bandit (MAB) problem in
which the distribution of rewards associated with each arm are assumed to be
time-varying and the total variation in the expected rewards is subject to a
variation budget. The regret of a policy is defined by the difference in the
expected cumulative rewards obtained using the policy and using an oracle that
selects the arm with the maximum mean reward at each time. We characterize the
performance of the proposed policies in terms of the worst-case regret, which
is the supremum of the regret over the set of reward distribution sequences
satisfying the variation budget. We extend Upper-Confidence Bound (UCB)-based
policies with three different approaches, namely, periodic resetting, sliding
observation window and discount factor and show that they are order-optimal
with respect to the minimax regret, i.e., the minimum worst-case regret
achieved by any policy. We also relax the sub-Gaussian assumption on reward
distributions and develop robust versions the proposed polices that can handle
heavy-tailed reward distributions and maintain their performance guarantees.
