The Best of Both Worlds: Reinforcement Learning with Logarithmic Regret
  and Policy Switches
  In this paper, we study the problem of regret minimization for episodic
Reinforcement Learning (RL) both in the model-free and the model-based setting.
We focus on learning with general function classes and general model classes,
and we derive results that scale with the eluder dimension of these classes. In
contrast to the existing body of work that mainly establishes
instance-independent regret guarantees, we focus on the instance-dependent
setting and show that the regret scales logarithmically with the horizon $T$,
provided that there is a gap between the best and the second best action in
every state. In addition, we show that such a logarithmic regret bound is
realizable by algorithms with $O(\log T)$ switching cost (also known as
adaptivity complexity). In other words, these algorithms rarely switch their
policy during the course of their execution. Finally, we complement our results
with lower bounds which show that even in the tabular setting, we cannot hope
for regret guarantees lower than $o(\log T)$.
