Accelerating Reinforcement Learning through GPU Atari Emulation
  We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari
Learning Environment (ALE) which is used for the development of deep
reinforcement algorithms. CuLE overcomes many limitations of existing CPU-based
emulators and scales naturally to multiple GPUs. It leverages GPU
parallelization to run thousands of games simultaneously and it renders frames
directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU
communication bandwidth. CuLE generates up to 155M frames per hour on a single
GPU, a finding previously achieved only through a cluster of CPUs. Beyond
highlighting the differences between CPU and GPU emulators in the context of
reinforcement learning, we show how to leverage the high throughput of CuLE by
effective batching of the training data, and show accelerated convergence for
A2C+V-trace. CuLE is available at https://github.com/NVLabs/cule .
