GAN-Based Interactive Reinforcement Learning from Demonstration and
  Human Evaluative Feedback
  Deep reinforcement learning (DRL) has achieved great successes in many
simulated tasks. The sample inefficiency problem makes applying traditional DRL
methods to real-world robots a great challenge. Generative Adversarial
Imitation Learning (GAIL) -- a general model-free imitation learning method,
allows robots to directly learn policies from expert trajectories in large
environments. However, GAIL shares the limitation of other imitation learning
methods that they can seldom surpass the performance of demonstrations. In this
paper, to address the limit of GAIL, we propose GAN-Based Interactive
Reinforcement Learning (GAIRL) from demonstration and human evaluative feedback
by combining the advantages of GAIL and interactive reinforcement learning. We
tested our proposed method in six physics-based control tasks, ranging from
simple low-dimensional control tasks -- Cart Pole and Mountain Car, to
difficult high-dimensional tasks -- Inverted Double Pendulum, Lunar Lander,
Hopper and HalfCheetah. Our results suggest that with both optimal and
suboptimal demonstrations, a GAIRL agent can always learn a more stable policy
with optimal or close to optimal performance, while the performance of the GAIL
agent is upper bounded by the performance of demonstrations or even worse than
it. In addition, our results indicate the reason that GAIRL is superior over
GAIL is the complementary effect of demonstrations and human evaluative
feedback.
