Generalized Proximal Policy Optimization with Sample Reuse
  In real-world decision making tasks, it is critical for data-driven
reinforcement learning methods to be both stable and sample efficient.
On-policy methods typically generate reliable policy improvement throughout
training, while off-policy methods make more efficient use of data through
sample reuse. In this work, we combine the theoretically supported stability
benefits of on-policy algorithms with the sample efficiency of off-policy
algorithms. We develop policy improvement guarantees that are suitable for the
off-policy setting, and connect these bounds to the clipping mechanism used in
Proximal Policy Optimization. This motivates an off-policy version of the
popular algorithm that we call Generalized Proximal Policy Optimization with
Sample Reuse. We demonstrate both theoretically and empirically that our
algorithm delivers improved performance by effectively balancing the competing
goals of stability and sample efficiency.
