Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in
  MDPs
  The problem of reinforcement learning in an unknown and discrete Markov
Decision Process (MDP) under the average-reward criterion is considered, when
the learner interacts with the system in a single stream of observations,
starting from an initial state without any reset. We revisit the minimax lower
bound for that problem by making appear the local variance of the bias function
in place of the diameter of the MDP. Furthermore, we provide a novel analysis
of the KL-UCRL algorithm establishing a high-probability regret bound scaling
as $\widetilde {\mathcal O}\Bigl({\textstyle \sqrt{S\sum_{s,a}{\bf
V}^\star_{s,a}T}}\Big)$ for this algorithm for ergodic MDPs, where $S$ denotes
the number of states and where ${\bf V}^\star_{s,a}$ is the variance of the
bias function with respect to the next-state distribution following action $a$
in state $s$. The resulting bound improves upon the best previously known
regret bound $\widetilde {\mathcal O}(DS\sqrt{AT})$ for that algorithm, where
$A$ and $D$ respectively denote the maximum number of actions (per state) and
the diameter of MDP. We finally compare the leading terms of the two bounds in
some benchmark MDPs indicating that the derived bound can provide an order of
magnitude improvement in some cases. Our analysis leverages novel variations of
the transportation lemma combined with Kullback-Leibler concentration
inequalities, that we believe to be of independent interest.
