Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
  We study the global convergence and global optimality of actor-critic, one of
the most popular families of reinforcement learning algorithms. While most
existing works on actor-critic employ bi-level or two-timescale updates, we
focus on the more practical single-timescale setting, where the actor and
critic are updated simultaneously. Specifically, in each iteration, the critic
update is obtained by applying the Bellman evaluation operator only once while
the actor is updated in the policy gradient direction computed using the
critic. Moreover, we consider two function approximation settings where both
the actor and critic are represented by linear or deep neural networks. For
both cases, we prove that the actor sequence converges to a globally optimal
policy at a sublinear $O(K^{-1/2})$ rate, where $K$ is the number of
iterations. To the best of our knowledge, we establish the rate of convergence
and global optimality of single-timescale actor-critic with linear function
approximation for the first time. Moreover, under the broader scope of policy
optimization with nonlinear function approximation, we prove that actor-critic
with deep neural network finds the globally optimal policy at a sublinear rate
for the first time.
