Diagnosing Bottlenecks in Deep Q-learning Algorithms
  Q-learning methods represent a commonly used class of algorithms in
reinforcement learning: they are generally efficient and simple, and can be
combined readily with function approximators for deep reinforcement learning
(RL). However, the behavior of Q-learning methods with function approximation
is poorly understood, both theoretically and empirically. In this work, we aim
to experimentally investigate potential issues in Q-learning, by means of a
"unit testing" framework where we can utilize oracles to disentangle sources of
error. Specifically, we investigate questions related to function
approximation, sampling error and nonstationarity, and where available, verify
if trends found in oracle settings hold true with modern deep RL methods. We
find that large neural network architectures have many benefits with regards to
learning stability; offer several practical compensations for overfitting; and
develop a novel sampling method based on explicitly compensating for function
approximation error that yields fair improvement on high-dimensional continuous
control domains.
