Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations
  using Generative Models
  The potential benefits of model-free reinforcement learning to real robotics
systems are limited by its uninformed exploration that leads to slow
convergence, lack of data-efficiency, and unnecessary interactions with the
environment. To address these drawbacks we propose a method that combines
reinforcement and imitation learning by shaping the reward function with a
state-and-action-dependent potential that is trained from demonstration data,
using a generative model. We show that this accelerates policy learning by
specifying high-value areas of the state and action space that are worth
exploring first. Unlike the majority of existing methods that assume optimal
demonstrations and incorporate the demonstration data as hard constraints on
policy optimization, we instead incorporate demonstration data as advice in the
form of a reward shaping potential trained as a generative model of states and
actions. In particular, we examine both normalizing flows and Generative
Adversarial Networks to represent these potentials. We show that, unlike many
existing approaches that incorporate demonstrations as hard constraints, our
approach is unbiased even in the case of suboptimal and noisy demonstrations.
We present an extensive range of simulations, as well as experiments on the
Franka Emika 7DOF arm, to demonstrate the practicality of our method.
