Policy Mirror Descent for Reinforcement Learning: Linear Convergence,
  New Sampling Complexity, and Generalized Problem Classes
  We present new policy mirror descent (PMD) methods for solving reinforcement
learning (RL) problems with either strongly convex or general convex
regularizers. By exploring the structural properties of these overall highly
nonconvex problems we show that the PMD methods exhibit fast linear rate of
convergence to the global optimality. We develop stochastic counterparts of
these methods, and establish an ${\cal O}(1/\epsilon)$ (resp., ${\cal
O}(1/\epsilon^2)$) sampling complexity for solving these RL problems with
strongly (resp., general) convex regularizers using different sampling schemes,
where $\epsilon$ denote the target accuracy. We further show that the
complexity for computing the gradients of these regularizers, if necessary, can
be bounded by ${\cal O}\{(\log_\gamma \epsilon) [(1-\gamma)L/\mu]^{1/2}\log
(1/\epsilon)\}$ (resp., ${\cal O} \{(\log_\gamma \epsilon )
(L/\epsilon)^{1/2}\}$)for problems with strongly (resp., general) convex
regularizers. Here $\gamma$ denotes the discounting factor. To the best of our
knowledge, these complexity bounds, along with our algorithmic developments,
appear to be new in both optimization and RL literature. The introduction of
these convex regularizers also greatly expands the flexibility and
applicability of RL models.
