Online Model Selection for Reinforcement Learning with Function
  Approximation
  Deep reinforcement learning has achieved impressive successes yet often
requires a very large amount of interaction data. This result is perhaps
unsurprising, as using complicated function approximation often requires more
data to fit, and early theoretical results on linear Markov decision processes
provide regret bounds that scale with the dimension of the linear
approximation. Ideally, we would like to automatically identify the minimal
dimension of the approximation that is sufficient to encode an optimal policy.
Towards this end, we consider the problem of model selection in RL with
function approximation, given a set of candidate RL algorithms with known
regret guarantees. The learner's goal is to adapt to the complexity of the
optimal algorithm without knowing it \textit{a priori}. We present a
meta-algorithm that successively rejects increasingly complex models using a
simple statistical test. Given at least one candidate that satisfies
realizability, we prove the meta-algorithm adapts to the optimal complexity
with $\tilde{O}(L^{5/6} T^{2/3})$ regret compared to the optimal candidate's
$\tilde{O}(\sqrt T)$ regret, where $T$ is the number of episodes and $L$ is the
number of algorithms. The dimension and horizon dependencies remain optimal
with respect to the best candidate, and our meta-algorithmic approach is
flexible to incorporate multiple candidate algorithms and models. Finally, we
show that the meta-algorithm automatically admits significantly improved
instance-dependent regret bounds that depend on the gaps between the maximal
values attainable by the candidates.
