Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep
  Reinforcement Learning with Demonstration-like Sampled Exploration
  In complex environments with high dimension, training a reinforcement
learning (RL) model from scratch often suffers from lengthy and tedious
collection of agent-environment interactions. Instead, leveraging expert
demonstration to guide RL agent can boost sample efficiency and improve final
convergence. In order to better integrate expert prior with on-policy RL
models, we propose a generic framework for Learning from Demonstration (LfD)
based on actor-critic algorithms. Technically, we first employ K-Means
clustering to evaluate the similarity of sampled exploration with demonstration
data. Then we increase the likelihood of actions in similar frames by modifying
the gradient update strategy to leverage demonstration. We conduct experiments
on 4 standard benchmark environments in Mujoco and 2 self-designed robotic
environments. Results show that, under certain condition, our algorithm can
improve sample efficiency by 20% ~ 40%. By combining our framework with
on-policy algorithms, RL models can accelerate convergence and obtain better
final mean episode rewards especially in complex robotic context where
interactions are expensive.
