Towards Understanding Asynchronous Advantage Actor-critic: Convergence
  and Linear Speedup
  Asynchronous and parallel implementation of standard reinforcement learning
(RL) algorithms is a key enabler of the tremendous success of modern RL. Among
many asynchronous RL algorithms, arguably the most popular and effective one is
the asynchronous advantage actor-critic (A3C) algorithm. Although A3C is
becoming the workhorse of RL, its theoretical properties are still not
well-understood, including its non-asymptotic analysis and the performance gain
of parallelism (a.k.a. linear speedup). This paper revisits the A3C algorithm
and establishes its non-asymptotic convergence guarantees. Under both i.i.d.
and Markovian sampling, we establish the local convergence guarantee for A3C in
the general policy approximation case and the global convergence guarantee in
softmax policy parameterization. Under i.i.d. sampling, A3C obtains sample
complexity of $\mathcal{O}(\epsilon^{-2.5}/N)$ per worker to achieve $\epsilon$
accuracy, where $N$ is the number of workers. Compared to the best-known sample
complexity of $\mathcal{O}(\epsilon^{-2.5})$ for two-timescale AC, A3C achieves
\emph{linear speedup}, which justifies the advantage of parallelism and
asynchrony in AC algorithms theoretically for the first time. Numerical tests
on synthetic environment, OpenAI Gym environments and Atari games have been
provided to verify our theoretical analysis.
