Proximal Policy Optimization with Mixed Distributed Training
  Instability and slowness are two main problems in deep reinforcement
learning. Even if proximal policy optimization (PPO) is the state of the art,
it still suffers from these two problems. We introduce an improved algorithm
based on proximal policy optimization, mixed distributed proximal policy
optimization (MDPPO), and show that it can accelerate and stabilize the
training process. In our algorithm, multiple different policies train
simultaneously and each of them controls several identical agents that interact
with environments. Actions are sampled by each policy separately as usual, but
the trajectories for the training process are collected from all agents,
instead of only one policy. We find that if we choose some auxiliary
trajectories elaborately to train policies, the algorithm will be more stable
and quicker to converge especially in the environments with sparse rewards.
