Non-local Policy Optimization via Diversity-regularized Collaborative
  Exploration
  Conventional Reinforcement Learning (RL) algorithms usually have one single
agent learning to solve the task independently. As a result, the agent can only
explore a limited part of the state-action space while the learned behavior is
highly correlated to the agent's previous experience, making the training prone
to a local minimum. In this work, we empower RL with the capability of teamwork
and propose a novel non-local policy optimization framework called
Diversity-regularized Collaborative Exploration (DiCE). DiCE utilizes a group
of heterogeneous agents to explore the environment simultaneously and share the
collected experiences. A regularization mechanism is further designed to
maintain the diversity of the team and modulate the exploration. We implement
the framework in both on-policy and off-policy settings and the experimental
results show that DiCE can achieve substantial improvement over the baselines
in the MuJoCo locomotion tasks.
