Never Forget: Balancing Exploration and Exploitation via Learning
  Optical Flow
  Exploration bonus derived from the novelty of the states in an environment
has become a popular approach to motivate exploration for deep reinforcement
learning agents in the past few years. Recent methods such as curiosity-driven
exploration usually estimate the novelty of new observations by the prediction
errors of their system dynamics models. Due to the capacity limitation of the
models and difficulty of performing next-frame prediction, however, these
methods typically fail to balance between exploration and exploitation in
high-dimensional observation tasks, resulting in the agents forgetting the
visited paths and exploring those states repeatedly. Such inefficient
exploration behavior causes significant performance drops, especially in large
environments with sparse reward signals. In this paper, we propose to introduce
the concept of optical flow estimation from the field of computer vision to
deal with the above issue. We propose to employ optical flow estimation errors
to examine the novelty of new observations, such that agents are able to
memorize and understand the visited states in a more comprehensive fashion. We
compare our method against the previous approaches in a number of experimental
experiments. Our results indicate that the proposed method appears to deliver
superior and long-lasting performance than the previous methods. We further
provide a set of comprehensive ablative analysis of the proposed method, and
investigate the impact of optical flow estimation on the learning curves of the
DRL agents.
