Planning with Exploration: Addressing Dynamics Bottleneck in Model-based
  Reinforcement Learning
  Model-based reinforcement learning (MBRL) is believed to have higher sample
efficiency compared with model-free reinforcement learning (MFRL). However,
MBRL is plagued by dynamics bottleneck dilemma. Dynamics bottleneck dilemma is
the phenomenon that the performance of the algorithm falls into the local
optimum instead of increasing when the interaction step with the environment
increases, which means more data can not bring better performance. In this
paper, we find that the trajectory reward estimation error is the main reason
that causes dynamics bottleneck dilemma through theoretical analysis. We give
an upper bound of the trajectory reward estimation error and point out that
increasing the agent's exploration ability is the key to reduce trajectory
reward estimation error, thereby alleviating dynamics bottleneck dilemma.
Motivated by this, a model-based control method combined with exploration named
MOdel-based Progressive Entropy-based Exploration (MOPE2) is proposed. We
conduct experiments on several complex continuous control benchmark tasks. The
results verify that MOPE2 can effectively alleviate dynamics bottleneck dilemma
and have higher sample efficiency than previous MBRL and MFRL algorithms.
