The Architectural Implications of Distributed Reinforcement Learning on
  CPU-GPU Systems
  With deep reinforcement learning (RL) methods achieving results that exceed
human capabilities in games, robotics, and simulated environments, continued
scaling of RL training is crucial to its deployment in solving complex
real-world problems. However, improving the performance scalability and power
efficiency of RL training through understanding the architectural implications
of CPU-GPU systems remains an open problem. In this work we investigate and
improve the performance and power efficiency of distributed RL training on
CPU-GPU systems by approaching the problem not solely from the GPU
microarchitecture perspective but following a holistic system-level analysis
approach. We quantify the overall hardware utilization on a state-of-the-art
distributed RL training framework and empirically identify the bottlenecks
caused by GPU microarchitectural, algorithmic, and system-level design choices.
We show that the GPU microarchitecture itself is well-balanced for
state-of-the-art RL frameworks, but further investigation reveals that the
number of actors running the environment interactions and the amount of
hardware resources available to them are the primary performance and power
efficiency limiters. To this end, we introduce a new system design metric,
CPU/GPU ratio, and show how to find the optimal balance between CPU and GPU
resources when designing scalable and efficient CPU-GPU systems for RL
training.
