Parallel Actors and Learners: A Framework for Generating Scalable RL
  Implementations
  Reinforcement Learning (RL) has achieved significant success in application
domains such as robotics, games and health care. However, training RL agents is
very time consuming. Current implementations exhibit poor performance due to
challenges such as irregular memory accesses and thread-level synchronization
overheads on CPU. In this work, we propose a framework for generating scalable
reinforcement learning implementations on multi-core systems. Replay Buffer is
a key component of RL algorithms which facilitates storage of samples obtained
from environmental interactions and data sampling for the learning process. We
define a new data structure for Prioritized Replay Buffer based on $K$-ary sum
tree that supports asynchronous parallel insertions, sampling, and priority
updates. To address the challenge of irregular memory accesses, we propose a
novel data layout to store the nodes of the sum tree that reduces the number of
cache misses. Additionally, we propose $\textit{lazy writing}$ mechanism to
reduce thread-level synchronization overheads of the Replay Buffer operations.
Our framework employs parallel actors to concurrently collect data via
environmental interactions, and parallel learners to perform stochastic
gradient descent using the collected data. Our framework supports a wide range
of reinforcement learning algorithms including DQN, DDPG, etc. We demonstrate
the effectiveness of our framework in accelerating RL algorithms by performing
experiments on CPU + GPU platform using OpenAI benchmarks.
