High-Throughput Synchronous Deep RL
  Deep reinforcement learning (RL) is computationally demanding and requires
processing of many data points. Synchronous methods enjoy training stability
while having lower data throughput. In contrast, asynchronous methods achieve
high throughput but suffer from stability issues and lower sample efficiency
due to `stale policies.' To combine the advantages of both methods we propose
High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). In HTS-RL, we
perform learning and rollouts concurrently, devise a system design which avoids
`stale policies' and ensure that actors interact with environment replicas in
an asynchronous manner while maintaining full determinism. We evaluate our
approach on Atari games and the Google Research Football environment. Compared
to synchronous baselines, HTS-RL is 2-6$\times$ faster. Compared to
state-of-the-art asynchronous methods, HTS-RL has competitive throughput and
consistently achieves higher average episode rewards.
