Is a Good Representation Sufficient for Sample Efficient Reinforcement
  Learning?
  Modern deep learning methods provide effective means to learn good
representations. However, is a good representation itself sufficient for sample
efficient reinforcement learning? This question has largely been studied only
with respect to (worst-case) approximation error, in the more classical
approximate dynamic programming literature. With regards to the statistical
viewpoint, this question is largely unexplored, and the extant body of
literature mainly focuses on conditions which permit sample efficient
reinforcement learning with little understanding of what are necessary
conditions for efficient reinforcement learning.
  This work shows that, from the statistical viewpoint, the situation is far
subtler than suggested by the more traditional approximation viewpoint, where
the requirements on the representation that suffice for sample efficient RL are
even more stringent. Our main results provide sharp thresholds for
reinforcement learning methods, showing that there are hard limitations on what
constitutes good function approximation (in terms of the dimensionality of the
representation), where we focus on natural representational conditions relevant
to value-based, model-based, and policy-based learning. These lower bounds
highlight that having a good (value-based, model-based, or policy-based)
representation in and of itself is insufficient for efficient reinforcement
learning, unless the quality of this approximation passes certain hard
thresholds. Furthermore, our lower bounds also imply exponential separations on
the sample complexity between 1) value-based learning with perfect
representation and value-based learning with a good-but-not-perfect
representation, 2) value-based learning and policy-based learning, 3)
policy-based learning and supervised learning and 4) reinforcement learning and
imitation learning.
