Revisiting Model-based Value Expansion
  Model-based value expansion methods promise to improve the quality of value
function targets and, thereby, the effectiveness of value function learning.
However, to date, these methods are being outperformed by Dyna-style algorithms
with conceptually simpler 1-step value function targets. This shows that in
practice, the theoretical justification of value expansion does not seem to
hold. We provide a thorough empirical study to shed light on the causes of
failure of value expansion methods in practice which is believed to be the
compounding model error. By leveraging GPU based physics simulators, we are
able to efficiently use the true dynamics for analysis inside the model-based
reinforcement learning loop. Performing extensive comparisons between true and
learned dynamics sheds light into this black box. This paper provides a better
understanding of the actual problems in value expansion. We provide future
directions of research by empirically testing the maximum theoretical
performance of current approaches.
