Memory-Constrained Policy Optimization
  We introduce a new constrained optimization method for policy gradient
reinforcement learning, which uses two trust regions to regulate each policy
update. In addition to using the proximity of one single old policy as the
first trust region as done by prior works, we propose to form a second trust
region through the construction of another virtual policy that represents a
wide range of past policies. We then enforce the new policy to stay closer to
the virtual policy, which is beneficial in case the old policy performs badly.
More importantly, we propose a mechanism to automatically build the virtual
policy from a memory buffer of past policies, providing a new capability for
dynamically selecting appropriate trust regions during the optimization
process. Our proposed method, dubbed as Memory-Constrained Policy Optimization
(MCPO), is examined on a diverse suite of environments including robotic
locomotion control, navigation with sparse rewards and Atari games,
consistently demonstrating competitive performance against recent on-policy
constrained policy gradient methods.
