Adaptive Discretization in Online Reinforcement Learning
  Discretization based approaches to solving online reinforcement learning
problems have been studied extensively in practice on applications ranging from
resource allocation to cache management. Two major questions in designing
discretization-based algorithms are how to create the discretization and when
to refine it. While there have been several experimental results investigating
heuristic solutions to these questions, there has been little theoretical
treatment. In this paper we provide a unified theoretical analysis of
tree-based hierarchical partitioning methods for online reinforcement learning,
providing model-free and model-based algorithms. We show how our algorithms are
able to take advantage of inherent structure of the problem by providing
guarantees that scale with respect to the 'zooming dimension' instead of the
ambient dimension, an instance-dependent quantity measuring the benignness of
the optimal $Q_h^\star$ function.
  Many applications in computing systems and operations research requires
algorithms that compete on three facets: low sample complexity, mild storage
requirements, and low computational burden. Our algorithms are easily adapted
to operating constraints, and our theory provides explicit bounds across each
of the three facets. This motivates its use in practical applications as our
approach automatically adapts to underlying problem structure even when very
little is known a priori about the system.
