Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon
  Reinforcement Learning?
  Learning to plan for long horizons is a central challenge in episodic
reinforcement learning problems. A fundamental question is to understand how
the difficulty of the problem scales as the horizon increases. Here the natural
measure of sample complexity is a normalized one: we are interested in the
number of episodes it takes to provably discover a policy whose value is
$\varepsilon$ near to that of the optimal value, where the value is measured by
the normalized cumulative reward in each episode. In a COLT 2018 open problem,
Jiang and Agarwal conjectured that, for tabular, episodic reinforcement
learning problems, there exists a sample complexity lower bound which exhibits
a polynomial dependence on the horizon -- a conjecture which is consistent with
all known sample complexity upper bounds. This work refutes this conjecture,
proving that tabular, episodic reinforcement learning is possible with a sample
complexity that scales only logarithmically with the planning horizon. In other
words, when the values are appropriately normalized (to lie in the unit
interval), this results shows that long horizon RL is no more difficult than
short horizon RL, at least in a minimax sense. Our analysis introduces two
ideas: (i) the construction of an $\varepsilon$-net for optimal policies whose
log-covering number scales only logarithmically with the planning horizon, and
(ii) the Online Trajectory Synthesis algorithm, which adaptively evaluates all
policies in a given policy class using sample complexity that scales with the
log-covering number of the given policy class. Both may be of independent
interest.
