Fully Asynchronous Policy Evaluation in Distributed Reinforcement
  Learning over Networks
  This paper proposes a \emph{fully asynchronous} scheme for the policy
evaluation problem of distributed reinforcement learning (DisRL) over directed
peer-to-peer networks. Without waiting for any other node of the network, each
node can locally update its value function at any time by using (possibly
delayed) information from its neighbors. This is in sharp contrast to the
gossip-based scheme where a pair of nodes concurrently update. Though the fully
asynchronous setting involves a difficult multi-timescale decision problem, we
design a novel stochastic average gradient (SAG) based distributed algorithm
and develop a push-pull augmented graph approach to prove its exact convergence
at a linear rate of $\mathcal{O}(c^k)$ where $c\in(0,1)$ and $k$ increases by
one no matter on which node updates. Finally, numerical experiments validate
that our method speeds up linearly with respect to the number of nodes, and is
robust to straggler nodes.
