Internal Model from Observations for Reward Shaping
  Reinforcement learning methods require careful design involving a reward
function to obtain the desired action policy for a given task. In the absence
of hand-crafted reward functions, prior work on the topic has proposed several
methods for reward estimation by using expert state trajectories and action
pairs. However, there are cases where complete or good action information
cannot be obtained from expert demonstrations. We propose a novel reinforcement
learning method in which the agent learns an internal model of observation on
the basis of expert-demonstrated state trajectories to estimate rewards without
completely learning the dynamics of the external environment from state-action
pairs. The internal model is obtained in the form of a predictive model for the
given expert state distribution. During reinforcement learning, the agent
predicts the reward as a function of the difference between the actual state
and the state predicted by the internal model. We conducted multiple
experiments in environments of varying complexity, including the Super Mario
Bros and Flappy Bird games. We show our method successfully trains good
policies directly from expert game-play videos.
