Non-Stationary Reinforcement Learning: The Blessing of (More) Optimism
  We consider un-discounted reinforcement learning (RL) in Markov decision
processes (MDPs) under temporal drifts, ie, both the reward and state
transition distributions are allowed to evolve over time, as long as their
respective total variations, quantified by suitable metrics, do not exceed
certain variation budgets. This setting captures the endogeneity, exogeneity,
uncertainty, and partial feedback in sequential decision-making scenarios, and
finds applications in vehicle remarketing and real-time bidding. We first
develop the Sliding Window Upper-Confidence bound for Reinforcement Learning
with Confidence Widening (SWUCRL2-CW) algorithm, and establish its dynamic
regret bound when the variation budgets are known. In addition, we propose the
Bandit-over-Reinforcement Learning (BORL) algorithm to adaptively tune the
SWUCRL2-CW algorithm to achieve the same dynamic regret bound, but in a
parameter-free manner, ie, without knowing the variation budgets. Finally, we
conduct numerical experiments to show that our proposed algorithms achieve
superior empirical performance compared to existing algorithms.
  Notably, the interplay between endogeneity and exogeneity presents a unique
challenge, absent in existing (stationary and non-stationary) stochastic online
learning settings, when we apply the conventional Optimism in Face of
Uncertainty principle to design algorithms with provably low dynamic regret for
RL in drifting MDPs. We overcome the challenge by a novel confidence widening
technique that incorporates additional optimism into our learning algorithms to
ensure low dynamic regret bounds. To extend our theoretical findings, we apply
our framework to inventory control problems, and demonstrate how one can
alternatively leverage special structures on the state transition distributions
to bypass the difficulty in exploring time-varying environments.
