Block Policy Mirror Descent
  In this paper, we present a new class of policy gradient (PG) methods, namely
the block policy mirror descent (BPMD) methods for solving a class of
regularized reinforcement learning (RL) problems with (strongly) convex
regularizers. Compared to the traditional PG methods with a batch update rule,
which visits and updates the policy for every state, BPMD methods have cheap
per-iteration computation via a partial update rule that performs the policy
update on a sampled state. Despite the nonconvex nature of the problem and a
partial update rule, BPMD methods achieve fast linear convergence to the global
optimality. We further extend BPMD methods to the stochastic setting, by
utilizing stochastic first-order information constructed from samples. We
establish $\mathcal{O}(1/\epsilon)$ (resp. $\mathcal{O}(1/\epsilon^2)$) sample
complexity for the strongly convex (resp. non-strongly convex) regularizers,
where $\epsilon$ denotes the target accuracy, with different procedures for
constructing the stochastic first-order information. To the best of our
knowledge, this is the first time that block coordinate descent methods have
been developed and analyzed for policy optimization in reinforcement learning,
which provides a new perspective on solving large-scale RL problems.
