Analysis of a Target-Based Actor-Critic Algorithm with Linear Function
  Approximation
  Actor-critic methods integrating target networks have exhibited a stupendous
empirical success in deep reinforcement learning. However, a theoretical
understanding of the use of target networks in actor-critic methods is largely
missing in the literature. In this paper, we reduce this gap between theory and
practice by proposing the first theoretical analysis of an online target-based
actor-critic algorithm with linear function approximation in the discounted
reward setting. Our algorithm uses three different timescales: one for the
actor and two for the critic. Instead of using the standard single timescale
temporal difference (TD) learning algorithm as a critic, we use a two
timescales target-based version of TD learning closely inspired from practical
actor-critic algorithms implementing target networks. First, we establish
asymptotic convergence results for both the critic and the actor under
Markovian sampling. Then, we provide a finite-time analysis showing the impact
of incorporating a target network into actor-critic methods.
